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Executive  Summary 


IDA  Paper  P-52181  examined  whether  changes  in  acquisition  policy  and  funding 
climate  had  statistically  discernible  associations  with  cancellations  of  Major  Defense 
Acquisition  Programs  (MDAPs).  Data  on  reductions  of  the  number  of  units  purchased — 
that  is,  truncations — of  programs  that  passed  Milestone  (MS)  B  post  fiscal  year  (FY) 
1988  also  were  presented,  but  not  examined  statistically.  This  paper  significantly  extends 
IDA  P-5218.  It  provides  parallel  analyses  of  both  cancellations  and  truncations,  has  more 
observations  (especially  on  truncations),  and  provides  an  improved  statistical  analysis  of 
cancellations. 

Brief  Census  of  Cancellations  and  Truncations 

The  paper  begins  with  a  brief  review  of  the  frequency  of  cancellations  and 
truncations,  which  is  not  part  of  the  common  knowledge  of  the  acquisition  community. 
The  database  used  contains  312  MDAPs  that  entered  Engineering  and  Management 
Development  (EMD)  during  FY  1965-FY  2009.  (MS  II,  which  is  now  designated  MS  B, 
was  not  established  until  1969.)  Of  these,  58  were  cancelled.  (See  table  below.)  The 
cancellation  rate  for  Joint  Programs  (26  percent)  is  somewhat  higher  than  the  average  for 
Service-managed  programs  (about  17  percent).  The  Department  of  the  Navy  has  the 
lowest  cancellation  rate  (14  percent)  and  the  Army  the  highest  (23  percent).  These 
differences  are  not  statistically  significant,  however.  Data  presented  in  the  main  text 
indicate  that  a  little  more  than  one-third  of  cancellations  apparently  were  initiated  by  a 
Military  Department. 

Cancellations,  Total  Programs,  and  Cancellation  Rate 
by  Military  Department  and  Joint  Programs 

No.  of  Cancellations  No.  of  Programs  Cancellation  Rate 


Army 

18 

77 

23% 

Navy 

15 

110 

14% 

Air  Force 

13 

79 

16% 

Joint 

12 

46 

26% 

Total 

58 

312 

19% 

1  David  L.  McNicol,  Sarah  K.  Burns,  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy 
and  Process  and  Funding  Climate  on  Cancellations  of  Major  Defense  Acquisition  Programs,”  IDA 
Paper  P-5218  (Alexandria,  VA:  Institute  for  Defense  Analyses,  May  2015). 
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We  also  computed  completion  ratios  (i.e.,  the  fraction  of  the  MS  B  baseline 
quantity  that  was  actually  bought)  for  162  completed  MDAPs.  The  median  program 
completed  100  percent  and  the  average  program  completed  118  percent  of  the  MS  B 
baseline  quantity.  About  60  percent  of  the  MDAPs  in  the  sample  completed  at  least  90 
percent  of  their  MS  B  baseline  quantity.  These  figures  are  somewhat  higher  than  is 
commonly  appreciated  and  serve  to  limit  the  extent  to  which  truncations  reasonably  can 
be  viewed  as  a  major  problem.  Twenty-four  MDAPs — about  one  in  seven  of  the  total 
sample — completed  less  than  50  percent  of  their  MS  B  baseline  quantity,  however.  Some 
of  these  would  fit  comfortably  on  a  list  of  cancelled  programs  but  others  would  not.  The 
F-22,  for  example,  acquired  only  29  percent  of  its  MS  B  quantity  and  the  F-14D  only  18 
percent.  In  short,  there  does  not  seem  to  be  any  clear  line  that  separates  truncations  that 
are  essentially  cancellations  from  those  that  are  not. 

Acquisition  Policy  and  Process,  Cancellations,  and  Truncations 

This  paper  recognizes  five  acquisition  policy  and  process  configurations  and  two 
funding  climates — bust  and  boom.  One  of  the  two  central  questions  of  the  paper  is 
whether,  for  a  given  funding  climate,  the  odds  that  an  MDAP  will  eventual  be  cancelled 
or  substantially  truncated  differ  significantly  depending  on  the  acquisition  process  and 
policy  in  effect  when  it  passed  MS  B.2 

A  significantly  higher  proportion  of  the  MDAPs  that  passed  MS  B  during  FY  1 987— 
FY  1989  were  cancelled.  This  appears  to  be  explained  primarily  by  unique  historical 
factors,  however,  rather  than  by  features  of  the  acquisition  policy  and  process  in  place  at 
that  time — in  particular,  the  attempt  of  the  Reagan  Administration  to  maintain  defense 
procurement  at  a  high  level.  While  FY  1987-FY  1989  may  be  a  partial  exception, 
generally  there  was  no  statistical  association  of  the  frequency  of  cancellations  and 
acquisition  policy  and  process  within  either  bust  or  boom  climates.  Similarly,  the  odds 
that  a  program  acquired  less  than  75  percent  of  its  MS  B  baseline  quantity  were  not 
associated  with  the  acquisition  policy  and  processes  and  funding  climate  when  it  passed 
MS  B. 

The  odd  thing  about  these  results  is  the  absence  of  any  reduction  in  cancellations  or 
increase  in  completion  ratios  associated  with  the  Packard  reforms  instituted  in  mid- 1969. 
Loosely,  unit  cost  growth  went  down  when  the  Packard  reforms  were  instituted  and 
stayed  down  until  the  Acquisition  Refonn  period  (FY  1994-FY  2000);  cancellation  rates 


2  P-5330  (Revised)  found  that  there  were  two  periods  (FY  1965-FY  1969  and  FY  2004-FY  2009)  during 
which  average  unit  cost  growth  was  significantly  higher  than  in  other  periods.  David  L.  McNicol  et  at, 
“Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of  Major  Defense  Acquisition 
Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for  Defense  Analyses,  August 
2016). 
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did  not.  This  is  odd  because  on  the  face  of  the  matter  similar  factors  govern  unit  cost 
growth  as  govern  the  likelihood  of  cancellation.  One  possible  resolution  of  this  apparent 
dilemma  is  that  some  “good”  programs  have  “bad”  (that  is,  unreasonably  optimistic)  cost 
estimates.  Another  possibility  is  that  many  cancellations  are  associated  with  funding 
declines,  not  program  failures. 

Funding  Climate,  Cancellations,  and  Truncations 

The  second  major  question  considered  in  this  paper  is  whether  cancellations  and 
truncations  are  associated  with  funding  climate,  for  a  given  acquisition  policy  and 
process  configuration. 

Neither  cancellations  nor  truncations  proved  to  be  statistically  associated  with  the 
funding  climate  that  prevailed  when  programs  passed  MS  B.  Cancellations  were, 
however,  associated  with  periods  of  sharp  declines  in  Department  of  Defense  (DoD) 
procurement  funding.  The  figure  below  shows  the  number  of  MDAPs  cancelled  in  each 
fiscal  year  over  the  period  FY  1965-FY  2016.  Two  clusters  of  cancellations  are  evident. 
The  first  of  these,  FY  1986-FY  1994,  was  the  start  of  the  bust  phase  following  the 
Carter-Reagan  boom.  From  peak  procurement  Budget  Authority  of  $183.9  billion  (FY 
2017  dollars)  in  FY  1985,  procurement  fell  to  $64.8  billion  in  FY  1994,  about  35  percent 
of  its  FY  1985  level.  The  second  period  of  declining  funding  was  FY  2009-FY  2013.  FY 
2013  DoD  procurement  funding  in  constant  2017  dollars  stood  at  about  55  percent  of  its 
FY  2008  level. 
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Twenty-one  MDAPs  were  cancelled  during  FY  1986-FY  1994  and  11  were 
cancelled  during  FY  2009-FY  2013.  During  the  14  years  of  these  two  periods  taken 
together,  32  MDAPs  were  cancelled,  an  average  of  nearly  2.3  cancellations  each  year. 
During  the  other  38  years,  on  average,  about  two  MDAPs  were  cancelled  every  three 
years  (for  a  total  of  26).  A  simple  model  was  used  to  compute  the  probabilities  of  the 
observed  cancellations  for  the  two  periods  of  declining  procurement  funding.  Using  that 
model,  the  probability  of  observing  21  or  more  cancellations  during  the  nine  years  of 
FY  1986-FY  1994  is  0.003.  The  corresponding  probability  of  observing  11  or  more 
cancellations  for  the  FY  2009-FY  2013  period  was  0.04.  In  contrast,  truncations  (to  less 
than  75  percent  of  the  MS  B  baseline  quantity)  did  not  show  a  tendency  to  cluster  in 
periods  of  sharply  decreasing  procurement  funding. 

Cost  Growth  and  Cancellations  and  Truncations 

Finally,  the  paper  takes  up  the  conventional  wisdom  connecting  high  cost  growth  to 
MDAP  cancellations.  Program  Acquisition  Unit  Cost  (PAUC)  includes  both  Research, 
Development,  Test,  and  Evaluation  (RDT&E)  and  procurement  cost,  and  probably  is  the 
relevant  measure  of  unit  cost  in  the  context  of  cancellations.  The  MS  B  baseline  value  for 
PAUC  can  be  thought  of  as  a  goal  or  a  prediction.  PAUC  growth  is  computed  by 
comparing  baseline  value  to  the  actual  PAUC  reported  in  the  program’s  last  Selected 
Acquisition  Report  (SAR).  Both  the  MS  B  PAUC  and  that  from  the  final  SAR  are  stated 
in  constant  dollars.  In  addition,  the  PAUC  from  the  final  SAR  is  normalized  to  the  MS  B 
quantity.  In  what  follows,  “quantity  nonnalized  PAUC  growth”  will  be  abbreviated  to 
simply  “PAUC  growth.”  For  the  purposes  of  this  paper,  high  unit  cost  growth  is 
arbitrarily  defined  as  PAUC  growth  against  the  MS  B  baseline  of  at  least  50  percent. 

The  database  used  in  this  research  contains  an  estimate  of  PAUC  for  156  completed 
MDAPs.  Of  these,  43  had  PAUC  growth  of  at  least  50  percent.  The  average  PAUC 
growth  of  these  programs  was  93  percent,  and  PAUC  at  least  doubled  for  9  of  the  43. 
Clearly,  then,  not  all  MDAPs  with  high  cost  growth  were  cancelled. 

Furthennore,  from  the  limited  information  available,  it  does  not  appear  that  all 
MDAPs  that  were  cancelled  had  high  unit  cost  growth.  It  proved  to  be  possible  to  get  an 
estimate  of  the  amount  of  PAUC  growth  that  occurred  prior  to  cancellation  for  25  of  the 
58  cancelled  MDAPs.  Only  10  of  those  25  cancelled  programs  showed  high  cost  growth. 
Another  10  had  PAUC  growth  of  between  zero  and  30  percent,  and  five  showed  negative 
PAUC  growth.  The  conclusion  from  these  estimates  then,  clearly,  would  be  that  not  all 
cancelled  programs  had  high  cost  growth.  The  PAUC  growth  estimates,  unfortunately, 
understate  the  true  PAUC  growth  for  cancelled  programs  in  that  they  do  not  capture  the 
cost  growth  between  the  date  of  the  SAR  used  and  the  tennination  of  the  program.  In 
addition,  there  tends  to  be  some  delay  in  reporting  cost  growth  that  can  be  expected  to 
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occur  based  on  the  evidence  to  date  but  which  has  not  yet  in  fact  materialized.  The  PAUC 
growth  estimates  for  the  25  programs,  then,  are  indicative  but  not  conclusive. 


Conclusions 

The  clearest  conclusion  offered  by  this  paper  is  that  cancellations  of  MDAPs  are 
concentrated  in  the  two  periods  during  which  DoD  procurement  funding  was  declining, 
often  sharply.  This  is  not  a  surprising  conclusion,  but  it  is  useful  to  know  that 
cancellation  of  major  programs  is  in  fact  one  of  the  ways  that  DoD  responded  to  large 
funding  reductions.  Truncations,  in  contrast,  were  not  clustered  in  periods  of  declining 
procurement  funding,  which  is  surprising. 

Apart  from  the  association  of  cancellations  and  decreases  in  procurement  funding, 
the  conclusions  offered  are  about  what  appears  not  to  matter — or  to  matter  much — in 
cancellation  and  truncation  decisions: 

•  Possibly  with  the  partial  exception  of  FY  1987-FY  1989,  within  a  funding 
climate  there  is  no  statistical  association  between  cancellations  or  completion 
ratios  of  less  than  75  percent  and  the  acquisition  policy  and  process  prevailing 
when  a  program  passes  MS  B. 

•  Second,  given  acquisition  policy  and  process  configuration,  there  is  no  statistical 
association  between  funding  climate  at  MS  B  and  cancellations  or  completion 
ratios  of  less  than  75  percent. 

•  Third,  while  cancelled  programs  on  average  had  higher  cost  growth  than 
programs  not  cancelled,  most  programs  with  PAUC  growth  of  more  than  50 
percent  were  not  cancelled.  Clearly,  there  is  more  behind  cancellations  than  cost 
growth.  The  evidence  does  not  pennit  a  similar  conclusion  for  truncations, 
although  it  pennits  those  so  inclined  to  accept  that  higher  PAUC  growth  tends  to 
result  in  lower  completion  rates. 

Is  it  reasonable  to  infer  from  these  conclusions  that  large  decreases  in  DoD  funding 
such  as  those  of  FY  1986-FY  1993  and  FY  2009-FY  2013  caused  the  cancellation  of 
MDAPs  that  otherwise  would  have  been  completed?  To  be  clear  about  this,  it  is 
necessary  to  distinguish  between  decisions  to  reduce  the  overall  DoD  acquisition 
portfolio  and  decisions  about  which  programs  to  cancel.  The  latter  involve  a  range  of 
considerations — how  well  the  programs  are  doing,  how  important  they  are,  and  the 
continued  salience  of  the  threats  to  which  they  respond,  among  others.  The  decision  on 
the  extent  to  which  cancellations  should  be  used  to  close  a  funding  gap  involves  choices 
among  bad  alternatives — cancellations,  delays  in  new  starts,  stretches  of  existing 
programs,  acceptance  of  less  capable  alternatives  in  some  cases,  and  adoption  of  very 
optimistic  costing  and  programmatic  decisions  for  both  new  starts  and — to  the  extent 
possible — ongoing  programs.  Viewed  from  this  angle,  the  root  cause  of  many 
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cancellations  seems  to  be  a  mismatch  between  DoD’s  missions  and  functions,  its  force 
structure,  and  its  funding. 
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A.  Introduction 

IDA  Paper  P-52181  examined  whether  changes  in  acquisition  policy  and  funding 
climate  had  statistically  discernible  associations  with  cancellations  of  Major  Defense 
Acquisition  Programs  (MDAPs).  Data  on  reductions  of  the  number  of  units  purchased  of 
programs  that  passed  Milestone  (MS)  B  post  fiscal  year  (FY)  1988  also  were  presented, 
but  not  examined  statistically.  This  paper  significantly  extends  IDA  P-5218;  it  provides 
parallel  analyses  of  both  cancellations  and  truncations,  has  more  observations  (especially 
on  truncations),  and  uses  different  statistical  methods  in  the  analysis  of  cancellations. 

Sections  C  through  E  examine  these  questions  in  turn: 

•  Is  there  a  statistical  association  between  changes  in  acquisition  policy  and 
process  (given  funding  climate)  and  the  frequency  of  cancellations  and 
truncations? 

•  Are  changes  in  Department  of  Defense  (DoD)  procurement  funding  associated 
with  cancellations  and  truncations  (given  acquisition  policy  and  process)? 

•  Is  very  high  cost  growth  a  major  cause  of  most  cancellations  and  truncations? 

Readers  who  are  inclined  to  believe  that  cost  growth  is  a  dominant  factor  in  cancellations 
may  wish  to  start  with  Section  E.  Those  who  have  read  any  of  the  earlier  papers  in  this 
series  probably  will  be  more  at  home  reading  the  sections  in  the  order  stated. 

A  brief  survey  of  cancellations  and  truncations  of  MDAPs  is  provided  in  Section  B. 
Conclusions  reached  are  summarized  in  Section  F. 

B.  Brief  Census  of  Cancellations  and  Truncations 

The  database  used  for  this  research  contains  312  MDAPs  that  entered  Engineering 
and  Management  Development  (EMD)  during  FY  1965-FY  2009.  An  MDAP  was 
classified  as  cancelled  if: 

•  The  program  did  not  result  in  production  of  any  fully  configured  end  items,  or 

•  Any  fully  configured  end  items  produced  were  used  only  for  testing  and 
development. 

Application  of  this  definition  was  not  clear-cut  for  six  programs  that  passed  MS  B  at 
the  Service  level,  later  filed  Selected  Acquisition  Reports  (SARs),  and  subsequently  were 
cancelled.  The  five  that  had  been  designated  as  an  Acquisition  Category  I  (ACAT  I) 


1  David  L.  McNicol,  Sarah  K.  Burns,  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy 
and  Process  and  Funding  Climate  on  Cancellations  of  Major  Defense  Acquisition  Programs,”  IDA 
Paper  P-5218  (Alexandria,  VA:  Institute  for  Defense  Analyses,  May  2015). 
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program  were  included  in  the  database  as  cancelled  programs  and  the  one  that  had  not 
become  an  AC  AT  I  program  was  excluded.2  In  addition,  four  MDAPs  that  fit  the 
definition  more  closely  than  not  were  counted  as  cancelled.3  Altogether,  58  of  the  312 
that  entered  EMD  during  FY  1965-FY  2009  were  classified  as  cancelled. 

Twelve  programs  that  filed  at  least  one  SAR  during  the  period  FY  1965-FY  2015 
but  were  not  designated  as  ACAT  I  and/or  did  not  pass  MS  B  also  were  cancelled.  These 
programs  are  not  in  the  database  or  included  in  the  list  of  cancelled  programs.  They  are, 
however,  listed  as  numbers  59-70  in  the  “Programs  Notes”  file  on  the  compact  disc  (CD) 
provided  in  a  pocket  on  the  inside  back  cover  of  this  paper. 

Table  1  presents  data  on  cancellations.  The  cancellation  rate  for  Joint  Programs  (26 
percent)  is  somewhat  higher  than  the  average  for  Service-managed  programs  (about  17 
percent).  The  Department  of  the  Navy  has  the  lowest  cancellation  rate  (14  percent)  and 
the  Army  the  highest  (23  percent).  These  differences  are  not  statistically  significant, 
however.4 


Table  1.  Cancellations,  Total  Programs,  and  Cancellation  Ratios 
by  Military  Department  and  Joint  Programs 


No.  of  Cancellations 

No.  of  Programs 

Cancellation  Rate 

Army 

18 

77 

23% 

Navy 

15 

110 

14% 

Air  Force 

13 

79 

16% 

Joint 

12 

46 

26% 

Total 

58 

312 

19% 

The  final  SAR  for  an  MDAP  that  has  been  cancelled  usually  identifies  (with  varying 
degrees  of  clarity)  who  initiated  the  cancellation.  For  example:  “President  Bush  ordered 
the  tennination  of  [SRAM  II]  on  27  Sept.  [19]91.”5  An  example  from  the  “less  clear”  end 
of  the  scale  is  provided  by  the  Joint  Ground  Launched  Tacit  Rainbow.  The  final  SAR  for 
the  program  states  that  its  funding  was  not  included  in  the  “FY92-FY93  President’s 


2  AN/WQR-Advanced  Deployable  System,  AQM-127A  Supersonic  Low  Altitude  Target,  Advanced  Seal 
Delivery  System,  ASM-135A  Air-Launched  Anti-Satellite  System,  and  Land  Warrior.  Extended  Range 
Munition  was  cancelled  before  it  was  designated  an  ACAT  I  program. 

3 

Roland,  Safeguard,  WIN-T  Inc.  3,  and  C-27J.  Brief  sketches  of  the  relevant  facts  are  provided  in  the 
Program  Notes  on  the  CD  included  with  this  paper. 

4  Chi-square,  P  =  0.179. 

5  SAR  for  SRAM  II,  December  31,  1991,7. 
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Budget”  and  that  the  program  was  cancelled  by  the  Secretary  of  Defense.6  Using  such 
statements  from  the  SARs  supplemented  by  materials  found  on  limited  searches  on  the 
internet,  the  initiative  for  each  cancellation  was  attributed  to  a  Military  Department 
(MilDep),  Office  of  the  Secretary  of  Defense  (OSD),  the  White  House,  or  the  Congress. 
In  those  cases  in  which  there  was  no  evidence  that  the  cancellation  was  initiated  by  the 
Congress,  the  White  House,  or  one  of  the  MilDeps,  responsibility  was  assigned  to  OSD. 

The  results  are  reported  in  Table  2.  It  is  worth  noting  that  a  little  more  than  one- 
third  of  cancellations  apparently  were  initiated  by  MilDeps.  A  cancellation,  however, 
may  actually  be  initiated  by  an  organization  other  than  the  one  formally  responsible  for  it. 
There  are  cases  in  which,  for  example,  a  MilDep  cancels  a  program  because  it  appears  to 
be  highly  likely  that  if  it  does  not  do  so,  OSD  or  the  Congress  will — in  which  case,  OSD 
or  the  Congress  probably  will  decide  how  to  reallocate  the  funding. 


Table  2.  Number  and  Proportion  of  Cancellations  Initiated  by 
Different  Levels  of  Government 


Government  Entity 

No.  of  Cancellations 

Proportion  of 
Cancellations 

Military  Department 

21 

36% 

OSD 

23 

40% 

White  House 

4 

7% 

Congress 

10 

17% 

Total 

58 

100% 

The  definition  of  cancellation  used  here  ensures  that  all  cancelled  programs  were  in 
EMD  or  in  the  early  stages  of  Low  Rate  Initial  Production  (LRIP).7  In  fact,  all  but  12 
were  in  EMD  and  only  one,  the  C-27J,  was  in  Full  Rate  Production.  The  average  time 
from  MS  B  to  cancellation  was  5.5  years,  and  half  of  all  cancellations  occurred  when  the 
program  was  no  more  than  4.8  years  beyond  MS  B,  as  shown  in  Figure  1.  The 
distribution  has  a  fairly  long  tail,  however.  One  program  was  cancelled  over  19  years 
after  it  passed  MS  B  and  two  others  were  cancelled  after  more  than  12  years. 


6  SAR  for  the  BGM-136  Joint  Ground  Launched  (JGL)  Tacit  Rainbow,  December  31,  1990,  4. 

7  Reforms  of  the  DoD  acquisition  system  introduced  by  David  Packard,  the  Deputy  Secretary  of  Defense, 
in  early  FY  1970  established  MSs  I,  II,  and  III  for  an  MDAP.  MS  II  approval  authorized  a  program  to 
enter  EMD;  MS  IIIA  authorized  LRIP.  A  revision  of  DoD  Instruction  (DoDI)  5000.2,  issued  October 
23,  2000,  formally  replaced  MSs  I,  II,  and  III  with  MSs  A,  B,  and  C.  The  definitions  are  such  that  MS  B 
is  placed  several  months  earlier  in  the  process  than  MS  II. 
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Figure  1.  Cumulative  Distribution  of  Time  between  MS  B  and  Cancellation 


The  discussion  now  turns  from  cancellations  to  truncations.  In  most  cases,  the  MS 
B  baseline  includes  a  statement  of  the  total  number  of  fully  configured  end  items  to  be 
acquired.  The  final  SAR  for  a  program  should  report  the  total  number  of  units  actually 
acquired.  Because  programs  that  acquire  more  than  the  MS  B  baseline  quantity  have  a 
negative  truncation  ratio  it  usually  is  preferable  to  refer  to  completion  ratios.  The 
completion  ratio  for  a  program  is  simply  the  total  number  of  end  items  acquired  divided 
by  the  anticipated  number  in  the  MS  B  baseline.8 

The  extent  to  which  a  program  has  acquired  less  or  more  of  its  MS  B  baseline 
quantity  can  be  gauged  reliably  only  after  the  program  has  ended  because  the  size  of  the 
planned  buy  can  be  cut  one  year,  the  cut  restored  the  next  year,  and  the  quantity  increased 
beyond  the  MS  B  baseline  the  year  after  that.  For  this  reason  the  completion  ratios  used 
in  the  analysis  are  for  completed  programs  only. 

The  database  used  in  this  paper  contains  the  MS  B  baseline  quantity  and  the 
quantity  actually  acquired  for  162  completed  MDAPs  that  began  EMD  during  the  period 
FY  1965-FY  2009.  Figure  2  is  a  histogram  of  the  percentage  of  the  respective  MS  B 
quantities  acquired  by  these  programs.  The  median  program  acquired  100  percent  and  the 
average  program  acquired  118  percent  of  the  MS  B  baseline  quantity.  About  60  percent 
of  the  MDAPs  in  the  sample  acquired  at  least  90  percent  of  their  MS  B  baseline  quantity. 


The  total  number  of  units  acquired  includes  both  those  purchased  with  procurement  funds  and  those 
purchased  with  Research,  Development,  Test,  and  Evaluation  (RDT&E)  funds.  Some  art  may  be 
required  to  define  the  units  in  which  to  state  the  number  of  fully  configured  end  items  acquired. 
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These  figures  are  somewhat  higher  than  is  commonly  appreciated  and  serve  to  limit  the 
extent  to  which  truncations  reasonably  can  be  viewed  as  a  major  problem. 


Figure  2.  Histogram  of  the  Percentage  of  the  MS  B  Baseline  Quantity  Acquired  by  162 
Completed  MDAPs  that  Passed  MS  B  FY  1965-FY  2009 


Twenty-four  of  the  162  completed  programs  (about  one  in  seven)  acquired  less 
than  50  percent  of  their  MS  B  baseline  quantity.  Some  of  these  would  fit  comfortably  on 
a  list  of  cancelled  programs  but  others  would  not.  The  F-22,  for  example,  acquired  only 
29  percent  of  its  MS  B  quantity,  and  the  F-14D,  only  18  percent.  In  short,  there  does  not 
seem  to  be  any  line  that  separates  truncations  into  those  that  are  essentially  cancellations 
and  those  that  are  not. 


C.  Acquisition  Policy  and  Process,  Cancellations,  and  Truncations 

This  section  turns  to  the  question  of  whether,  having  approximately  normalized  for 
funding  climate,  changes  in  acquisition  policy  and  process  configuration  are  statistically 
associated  with  cancellations  and  completion  ratios. 

The  following  acquisition  policy  and  process  configurations  are  used  in  binning  the 
data  for  statistical  analyses: 

•  McNamara-Clifford,  FY  1964-FY  1969; 

•  Defense  Systems  Acquisition  Review  Council  (DSARC),  FY  1970-FY  1982; 

•  Post  Carlucci  DSARC  (P-C  DSARC),  FY  1 983-FY  1989; 
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•  Defense  Acquisition  Board  (DAB),  FY  1990-FY  1993  and  FY  2001-FY  2009; 

•  Acquisition  Refonn  (AR),  FY  1994-FY  2000. 

Two  funding  climates  also  are  distinguished — bust  and  boom.  The  acquisition 
configurations  and  funding  climates  are  described  in  Appendix  A.  Readers  who  have  read 
previous  papers  in  this  series  or  who  are  generally  familiar  with  the  OSD-level 
acquisition  process  and  various  acquisition  reform  efforts  can  use  Appendix  A 
selectively.  Others  may  wish  to  read  it  before  proceeding  with  the  main  text. 

P-5330  (Revised)9  offered  three  conclusions  about  the  association  of  changes  in 
acquisition  policy  and  process  and  growth  in  unit  cost10  for  MDAPs  that  passed  MS  B  in 
bust  funding  climates: 

1.  The  Packard  refonns  introduced  in  mid- 1969  (DSARC)  are  associated  with  a 
statistically  significant  and  persistent  reduction  in  quantity  normalized  Average 
Procurement  Unit  Cost  (APUC)  growth  (hereafter  APUC  growth). 

2.  The  Carlucci  Initiatives  (P-C  DSARC)  and  the  statutory  changes  associated  with 
the  Packard  Commission  Report  (1986)  and  the  Goldwater-Nicholls  Act 
evidentially  did  not  result  in  any  further  reduction  in  APUC  growth  (DAB, 

FY  1990-FY  1993,  FY  2001-FY  2009). 

3.  Compared  to  other  periods,  average  APUC  growth  was  significantly  higher  for 
MDAPs  that  began  development  during  McNamara-Clifford  and  the  AR  period. 

Three  of  the  five  configurations  also  were  in  place  during  boom  funding  climates,  but 
none  had  significantly  higher  or  lower  average  levels  of  APUC  growth  in  them. 

Cancellations  and  low  completion  ratios  would  be  expected  to  show  similar  patterns 
based  on  the  hypothesis  that  their  root  causes  overlap  substantially  with  those  of  cost 
growth.  If  the  expected  patterns  do  appear,  our  understanding  of  cost  growth  in  MDAPs 
is  reinforced.  A  failure  of  the  expected  patterns  to  appear,  however,  would  tend  to  call 
into  question  our  understanding  of  the  causes  of  cost  growth. 

Table  3  provides  data  on  the  number  of  cancellations  and  cohort  size  arrayed  by 
acquisition  configuration  and  funding  climate.  The  third  column  reports  the  number  of 
cancellations  and  cohort  size;  for  example,  the  database  contains  24  MDAPs  that  entered 
what  is  now  called  EMD  during  the  McNamara-Clifford  period,  of  which  three  were 


9  David  L.  McNicol  et  al.,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of 
Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  August  2016). 

10  Appendix  B  provides  a  detailed  description  of  how  the  quantity  normalized  APUC  growth  estimates 
were  made. 
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cancelled.  The  next  column  gives  the  probability  of  observing  that  many  cancellations  or 
more,  on  the  assumption  that  cancellations  are  independently  and  identically  distributed 
in  each  of  the  periods.  A  sufficiently  low  probability  leads  to  the  rejection  of  that 
assumption  for  the  bin  in  question.  The  logic  here  is  that  of  a  coin-flipping  trial:  if  heads 
comes  up  too  often,  we  reject  the  initial  presumption  that  the  coin  is  consistent  from  one 
period  to  the  next  and  conclude  that  it  has  changed.  The  last  column  gives  the  probability 
of  observing  the  actual  number  of  cancellations  or  fewer  assuming  consistency,  which 
provides  the  same  check  for  apparently  low  cancellation  frequencies  as  the  preceding 
column  does  for  apparently  high  ones. 


Table  3.  Cohort  Size,  Number  of  Cancellations,  and  Estimated  Probability  of  the  Number  of 
Cancellations  by  Acquisition  Configuration  for  Completed  MDAPs 

Cancellations  Probability  Probability 
Period  Fiscal  Years  k  x  >  k  x  <  k 


Bust  Funding  Climates 


McNamara-Clifford 

1965-1969 

12.5%  (3  of  24) 

0.845 

0.332 

DSARC 

1970-1980 

15.1%  (11  of  73) 

0.810 

0.287 

P-C  DSARC 

1987-1989 

40.7%  (11  of  27) 

0.006* 

0.998 

DAB 

1990-1993 

2001-2002 

12.9%  (6  of  31) 

0.518 

0.659 

AR 

1994-2000 

13.0%  (6  of  46) 

0.874 

0.233 

Boom  Funding  Climates 

DSARC 

1981-1982 

22.2%  (4  of  18) 

0.451 

0.757 

P-C  DSARC 

1983-1986 

10.0%  (5  of  50) 

0.971 

0.070 

DAB 

2003-2009 

27.9%  (12  of  43) 

0.098 

0.950 

Note:  Probabilities  were  computed  using  the  binomial  distribution,  assuming  the  average 
cancellation  rate  across  constellations  for  the  given  funding  climate. 

*  Significant  at  less  than  the  1 0  percent  level  when  adjusted  for  the  number  of  comparisons  made. 

Each  of  the  five  acquisition  policy  and  process  configurations  was  in  place  during  a 
bust  climate.  The  average  cancellation  rate  across  all  of  the  bust  climates  was  18.4 
percent  (37  of  201  programs).  Only  the  P-C  DSARC  had  a  cancellation  frequency 
significantly  above  or  below  the  average. 11  After  correction  for  the  number  of 
comparisons  made,  the  cancellation  frequencies  for  none  of  the  three  acquisition  policy 


11  Ten  outcomes  are  evaluated  for  the  bust  climates.  There  is  a  more  than  10  percent  chance  that  at  least 
one  of  these  would  by  chance  appear  to  be  statistically  different  from  the  population  mean  even  if  the 
true  mean  of  each  of  the  bins  were  the  population  mean.  The  Bonferroni  Correction  provides  a 
conservative  adjustment  of  the  critical  value  in  such  situations.  In  particular,  when  10  comparisons  are 
made,  an  upper  limit  for  the  critical  value  after  the  Bonferroni  Correction  for  significance  at  the  10 
percent  level  is  0.010;  i.e.,  0.10  divided  by  10. 
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and  process  configurations  that  operated  in  boom  climates  was  significantly  different 
from  the  boom  climates  mean  (18.9  percent).  Overall,  apart  from  the  P-C  DSARC  period, 
the  funding  climate  prevailing  at  MS  B  was  not  associated  with  the  odds  that  a  program 
would  eventually  be  cancelled. 

Table  4  provides  data  for  completion  ratios  that  parallel  those  on  cancellations  in 
Table  3.  The  line  was  drawn  at  a  completion  ratio  of  75  percent.  After  adjusting  for  the 
number  of  comparisons,  the  average  completion  rate  for  none  of  the  bins  differed 
significantly  from  the  climate  period  average.  The  results  for  completion  ratios  are  then 
consistent  with  those  for  cancellations;  that  is,  the  odds  that  a  program  will  acquire  less 
than  75  percent  of  its  MS  B  baseline  quantity  are  not  associated  with  the  acquisition 
policy  and  process  and  funding  climate  when  it  passed  MS  B. 


Table  4.  Cohort  Size,  Number  of  Programs  with  a  Completion  Ratio  of  Less  than  75 
Percent,  and  Estimated  Probability  of  the  Completion  Ratio  for  Completed  MDAPs 


Period 

Fiscal  Years 

Completion 

k 

Probability 
x>  k 

Probability 
x<  k 

Bust  Funding  Climates 

McNamara-Clifford 

1965-1969 

30.0%  (6  of  20) 

0.436 

0.743 

DSARC 

1970-1980 

28.1%  (10  of  48) 

0.848 

0.248 

PC  DSARC 

1987-1989 

53.9%  (7  of  13) 

0.032 

0.992 

DAB 

1990-1993 

2000-2001 

31.3%  (5  of  16) 

0.417 

0.775 

AR 

1994-2000 

11.8%  (2  of  17) 

0.961 

0.136 

Boom  Funding  Climates 

DSARC 

1981-1982 

42.9%  (3  of  7) 

0.244 

0.929 

PC  DSARC 

1983-1986 

28.6%  (8  of  28) 

0.400 

0.750 

DAB 

2003-2009 

7.6%  (1  of  13) 

0.976 

0.127 

Note:  Probabilities  were  computed  using  the  binomial  distribution  assuming  the  average 

cancellation  rate  across  constellations  for  the  given  funding  climate. 


The  odd  thing  about  this  result  and  the  corresponding  result  on  cancellations  is  the 
absence  of  any  reduction  in  cancellations  or  increase  in  completion  ratios  associated  with 
the  Packard  reforms.  Loosely,  unit  cost  growth  went  down  when  the  Packard  reforms 
were  instituted  and  stayed  down;  cancellation  rates  did  not.  This  is  odd  because,  as  noted 
earlier,  on  the  face  of  the  matter  similar  factors  govern  unit  cost  growth  as  govern  the 
likelihood  of  cancellation.  Unreasonably  ambitious  perfonnance  goals  established  at  MS 
B,  in  particular,  are  likely  to  result  in  schedule  slips,  cost  growth,  and  performance 
shortfalls.  Even  though  the  key  factor  in  a  decision  to  cancel  may  be  technical  failure,  we 
would  expect  such  failures  to  be  associated  with  cost  growth  and  schedule  slips. 
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One  possible  resolution  of  this  apparent  dilemma  is  that  some  “good”  programs 
have  “bad”  cost  estimates.  Another  possibility,  discussed  in  the  following  section,  is  that 
many  cancellations  are  associated  with  funding  declines,  not  program  failures. 

D.  Funding  Climate,  Cancellations,  and  Truncations 

This  section  takes  up  the  question  of  whether,  given  an  acquisition  policy  and 
process  configuration,  funding  climate  is  statistically  associated  with  cancellations  and 
completion  ratios.  This  question  is  motivated  by  the  significant  statistical  association 
between  quantity  normalized  Program  Acquisition  Unit  Cost  (PAUC)  growth  and  the 
funding  climate  that  prevailed  at  MS  B. 12 

By  way  of  background,  relevant  data  on  quantity  normalized  PAUC  growth 
(hereafter  PAUC  growth)  are  presented  in  Table  5.  (See  page  16  and  Appendix  B  for  a 
description  of  how  PAUC  growth  was  computed.)  Three  of  the  five  acquisition  policy 
and  process  configurations  were  in  place  in  both  bust  and  boom  funding  climates.  These 
provide  a  natural  “with  and  without”  experiment  on  the  effect  of  funding  climate  given 
acquisition  policy  and  process  configuration.  Average  PAUC  growth  is  significantly 
higher  in  each  of  the  acquisition  policy  and  process  configurations  for  programs  that 
passed  MS  B  in  a  bust  funding  climate.  Note  that  three  programs  were  excluded  from  the 
data  for  the  boom  portion  of  P-C  DSARC — T-45,  JSTARS,  and  C-17 — because  their 
relatively  high  cost  growth  probably  is  mainly  due  to  the  nature  of  the  contracts  used  to 
acquire  them. 13  If,  instead,  these  three  programs  are  included,  the  average  PAUC  growth 
for  programs  that  passed  MS  B  during  the  boom  portion  of  P-C  DSARC  is  20  percent, 
which  is  noticeably  but  not  statistically  smaller  than  the  average  for  programs  that  passed 
MS  B  during  the  bust  portion  of  P-C  DSARC  (34  percent). 


12 

~  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process  on 
Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5126  (Alexandria,  VA:  Institute 
for  Defense  Analyses,  September  2014). 

13 

David  L.  McNicol,  “Post-Milestone  B  Funding  Climate  and  Cost  Growth  in  Major  Defense  Acquisition 
Programs,”  IDA  Paper  P-8091  (Alexandria,  VA:  Institute  for  Defense  Analyses,  forthcoming,  6-7). 
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Table  5.  PAUC  Growth  for  Completed  Programs  in  Bust  and  Boom  Funding  Climates  for  a 
Given  Acquisition  Policy  and  Process  Configuration 

Bust  Boom  P-Valuef 


DSARC 

37%  (49) 

13%  (6) 

0.077* 

P-C  DSARC 

34%  (11) 

12%  (26) 

0.066* 

DAB 

40%  (15) 

2%  (11) 

0.002*** 

All  Other 

53%  (35) 

none 

n/a 

Total 

42%  (110) 

15%  (46) 

<  0.0001*’ 

t  M-W  U.  DSARC:  U  =  81 ,  m  =  49,  n2  =  6.  P-C  DSARC:  U  =  1 99,  m  =  26,  n2  =  1 1 ;  excludes  T-45, 
JSTARS,  and  C-1 7.  DAB:  U  =  23.5,  m  =  1 5,  n2  =  1 1 .  Total:  U  =  1 350.5,  m  =  1 1 0,  n2  =  46. 

*  Significant  at  less  than  the  10  percent  level. 

***  Significant  at  less  than  the  1  percent  level. 

1.  Cancellations  and  Truncations  by  MS  B  Cohorts 

The  question  is  whether  cancellations  and  completion  rates  show  a  similar  pattern. 
The  relevant  data  are  presented  in  Table  6.  Cancellation  frequencies  are  significantly 
different  between  the  two  climates  for  P-C  DSARC  and  DAB.  Note  that  the  differences 
go  in  opposite  directions — programs  that  passed  MS  B  during  the  boom  phase  of  P-C 
DSARC  had  a  significantly  lower  cancellation  rate  than  those  that  passed  MS  B  in  the 
bust  phase,  while  the  opposite  is  the  case  for  the  DAB  configuration.  There  is  then  no 
consistent  climate  effect.  Note  also  that  the  average  cancellation  frequency  for  all 
completed  programs  that  passed  MS  B  in  bust  periods  is  19.8  percent,  which  is  not 
significantly  different  from  the  22.5  percent  for  programs  that  passed  in  boom  climates. 


Table  6.  Cancellations  in  Bust  and  Boom  Funding  Climates  for  Completed  Programs  Given 
Acquisition  Policy  and  Process  Configuration 


Bust 

Boom 

P-Valuef 

DSARC 

15.1%  (11  of  73) 

22.2%  (4  of  18) 

0.486 

P-C  DSARC 

40.7%  (11  of  27) 

10.4%  (5  of  48) 

0.003*** 

DAB 

20.7%  (6  of  29) 

42.9%  (12  of  28) 

0.072* 

All  Other 

15.5%  (9  of  58) 

none 

n/a 

Total 

19.8%  (37  of  187) 

22.5%  (21  of  93) 

0.888 

t  DSARC:  FET;  P-C  DSARC  FET;  DAB:  Chi-square;  Total:  Chi-square. 

*  Significant  at  less  than  the  10  percent  level. 

***  Significant  at  less  than  the  1  percent  level. 

The  Bonferroni  Correction  is  not  required  in  this  situation  because  the  tests  used  make  a  single  comparison 
separately  for  each  of  the  acquisition  policy  and  process  configurations. 
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Table  7  reports  data  on  average  completion  ratios  for  programs  that  passed  MS  B  in 
boom  and  bust  phases  for  the  three  acquisition  policy  and  process  configurations.  Two  of 
the  configurations  (DSARC  and  P-C  DSARC)  show  significant  differences  between  the 
two  climates  but,  as  with  cancellations,  they  go  in  opposite  directions — DSARC  had  a 
lower  completion  ratio  for  programs  that  passed  MS  B  in  the  boom  portion,  while  for  P-C 
DSARC  the  programs  that  passed  in  the  bust  phase  had  the  lower  completion  ratio. 
Again,  at  the  total  level,  the  average  completion  ratio  is  not  significantly  different 
between  the  two  climates. 


Table  7.  Average  Completion  Ratios  in  Bust  and  Boom  Funding  Climates  for  Completed 
Programs  Given  Acquisition  Policy  and  Process  Configuration 

Bust  Boom  P-Valuef 


DSARC 

148%  (48) 

77%  (7) 

0.056* 

P-C  DSARC 

68%  (13) 

117%  (28) 

0.051* 

DAB 

116%  (16) 

122%  (13) 

0.757 

All  Other 

113%  (37) 

none 

n/a 

Total 

123%  (114) 

112%  (48) 

0.542 

t  M-W  U.  DSARC:  U  =  92,  m  =48,  n2  =  7;  P-C  DSARC:  U  =  112,  m  =28,  n2  =  13;  DAB:  U  =  111.5, 
ni  =  16,  n2  =  13;  Total:  U  =  2570.5,  m  =  114,  n2  =  48. 

*  Significant  at  less  than  the  10  percent  level. 

The  Bonferroni  Correction  is  not  required  in  this  situation  because  the  tests  used  make  a  single  comparison 
separately  for  each  of  the  acquisition  policy  and  process  configurations. 

Further  comment  on  the  pattern  of  cancellations  for  P-C  DSARC  and  DAB  is 
warranted.  The  cancellation  frequency  for  P-C  DSARC  is  significantly  lower  for  MDAPs 
that  passed  MS  B  during  its  boom  phase.  This  follows  the  pattern  observed  for  PAUC 
growth  and  is  consistent  with  the  character  of  the  Carlucci  Initiatives,  adopted  in 
FY  1982.  Among  other  things,  the  Carlucci  Initiatives  were  intended  to  tighten  up  the 
DSARC  process  and  to  help  ensure  that  DoD  did  not  start  more  programs  than  reasonably 
anticipated  funding  would  support. 

There  are,  however,  reasons  for  not  accepting  the  statistical  result  at  face  value. 
First,  the  higher  cancellation  frequency  of  programs  that  passed  MS  B  during  FY  1987— 
FY  1989  is  in  part  explained  by  the  end  of  the  Cold  War.14  Some  of  those  MDAPs  were 
later  cancelled  because,  in  the  light  of  altered  perceptions  of  the  threats,  they  no  longer 
seemed  to  be  of  enough  utility  to  justify  their  cost.  While  the  SARs  do  not  spell  out  the 


14  The  Berlin  Wall  fell  on  November  9,  1989,  just  over  one  month  into  FY  1990.  The  Soviet  Union  was 
formally  dissolved  in  April  1991. 
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fact,  there  is  little  doubt  that  changes  in  the  threat  were  a  major  factor  in  the  cancellations 
of  SRAM  II,  the  Small  ICBM,  Peacekeeper  Rail  Garrison,  and  possibly  some  other 
programs  from  this  cohort.  To  some  extent,  then,  the  high  cancellation  rate  of  the 
FY  1987-FY  1989  cohort  is  attributable  to  the  end  of  the  Cold  War  rather  than  to  the 
prevailing  acquisition  regime  and  funding  climate  at  MS  B. 

Second,  the  high  cancellation  frequency  for  the  FY  1987-FY  1989  cohort  also  may 
in  part  be  attributable  to  the  Administration  not  following  its  own  policy  on  new  starts. 
The  amount  appropriated  for  DoD  acquisition  fell  by  about  30  percent  from  FY  1986  to 
FY  1989,  and  with  the  passage  of  Gramm-Rudman-Hollings  (GRH)  and  developments  in 
Eastern  Europe,  there  was  no  reason  to  expect  an  increase  over  the  then  foreseeable 
future.  In  the  years  FY  1987-FY  1989,  however,  on  average  nine  programs  passed  MS  B 
annually.  This  rate  was  well  above  the  rate  for  other  bust  periods  (in  particular,  6.7/yr.  for 
FY  1970-FY  1980,  4/yr.  for  FY  1990-FY  1993,  and  6.6/yr.  for  FY  1994-FY  2000).  The 
relatively  high  new  start  rates  suggest  a  decreased  emphasis  on  the  Carlucci  Initiatives 
goal  of  not  starting  more  MDAPs  than  likely  future  budgets  could  sustain.  In  fact,  eight 
of  the  11  cancellations  from  the  new  starts  in  the  FY  1987-FY  1989  cohort  occurred 
during  FY  1990-FY  1993. 

Finally,  the  programs  that  passed  MS  B  during  the  post-9/11  boom  (DAB, 
FY  2003-FY  2009)  had  a  significantly  above  average  cancellation  frequency.  The  only 
readily  visible  reason  that  helps  to  explain  this  is  Secretary  of  Defense  Robert  Gates’ 
comparatively  high  willingness  to  cancel  programs.  (Gates  was  Secretary  from  December 
2006  until  July  2011.)  Cancellation  of  seven  MDAPs  was  clearly  attributed  to  a  Secretary 
of  Defense  in  the  relevant  SARs.  Gates  accounted  for  at  least  four,  and  probably  five,  of 
these.  The  four  cancellations  attributed  by  SARs  to  Gates  are  enough  to  account  for  the 
above  average  cancellation  rate  among  programs  that  passed  MS  B  during  FY  2003- 
FY  2009.  In  addition,  Gates  also  probably  initiated  one  cancellation  attributed  to  the 
President  and  two  that  the  SARs  attribute  to  OSD. 

The  upshot  of  these  comments  is  that,  while  the  differences  between  the  two 
climates  for  P-C  DSARC  and  DAB  are  significant,  their  explanation  probably  lies  outside 
the  operation  of  the  acquisition  process  during  the  relevant  periods. 

2,  Cancellations  and  Truncations  by  Fiscal  Year 

One  possible  explanation  starts  by  examining  whether  cancellations  are  associated 
with  sharp  declines  in  procurement  funding.  Pushed  to  the  limit,  the  possibility  raised  is 
that  in  many — perhaps  most — instances,  programs  are  cancelled  not  because  they  have 
failed,  but  to  close  a  funding  gap. 
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Figure  3  shows  the  number  of  cancellations  recorded  in  the  database  in  each  fiscal 
year  over  the  period  FY  1965-FY  2016.  Two  clusters  of  cancellations  are  evident  in  this 
figure,  one  centering  on  about  FY  1990  and  another  centering  on  about  FY  2010. 


There  were  in  fact  two  periods  during  which  DoD  procurement  appropriation 
decreased  from  one  year  to  the  next,  often  sharply.  The  first  of  these,  FY  1986-FY  1994, 
was  the  bust  phase  following  the  Carter-Reagan  boom  in  defense  budgets.  Funding  for 
DoD  procurement  fell  by  29  percent  in  constant  2017  dollars  over  the  last  five  budgets 
prepared  by  the  Reagan  Administration;  it  declined  by  an  additional  50  percent  over  the 
four  DoD  budgets  of  the  George  H.  W.  Bush  Administration.  From  peak  procurement 
Budget  Authority  of  $183.9  billion  (FY  2017  dollars)  in  FY  1985,  procurement  fell  to 
$64.8  billion  in  FY  1994,  about  35  percent  of  its  FY  1985  level.1''  Twenty-one  MDAPs 
were  cancelled  during  FY  1986-FY  1994. 

The  second  period  of  declining  funding  was  FY  2009-FY  2013.  The  financial  crisis 
that  sparked  the  Great  Recession  occurred  towards  the  end  of  Calendar  Year  (CY)  2008, 
or  approximately  during  the  first  quarter  of  FY  2009.  The  withdrawal  of  US  troops  from 
Iraq  began  in  July  2009  and  the  Budget  Control  Act  (BCA)  was  signed  into  law  late  in 
FY  2011.  FY  2013  DoD  procurement  funding  in  constant  2017  dollars  stood  at  about  55 
percent  of  its  FY  2008  level.  Eleven  MDAPs  were  cancelled  during  FY  2009-FY  2013. 


15  These  data,  as  well  as  similar  funding  data  cited  later,  are  in  billions  of  constant  FY  2017  dollars  of 
Budget  Authority.  They  are  from  Table  6-8  (p.  133ff)  of  the  Office  of  the  Under  Secretary  of  Defense 
(Comptroller),  National  Defense  Budget  Estimates  for  FY  201 7. 
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Thirty-two  MDAPs  were  cancelled  during  the  14  years  of  these  two  periods  taken 
together,  an  average  of  nearly  2.3  cancellations  each  year.  During  the  other  38  years,  on 
average,  about  two  MDAPs  were  cancelled  every  three  years.  A  simple  model  was  used 
to  compute  the  probabilities  of  the  observed  cancellations  for  these  two  periods.16  Using 
that  model,  the  probability  of  observing  21  or  more  cancellations  during  the  nine  years 
FY  1986-FY  1994  is  0.003.  The  corresponding  probability  of  observing  11  or  more 
cancellations  for  the  FY  2009-FY  2013  period  was  0.04. 

The  absence  of  a  cluster  of  cancellations  during  the  early  to  mid-1970s  may  be 
surprising  because  this  apparently  was  the  bust  phase  of  a  boom-bust  cycle  associated 
with  US  involvement  in  the  War  in  Vietnam.  Withdrawal  of  US  forces  from  Vietnam 
began  in  1969.  Virtually  all  US  forces  had  been  withdrawn  by  the  end  of  1972,  but  US 
material  support  for  South  Vietnam  continued  into  1975.  In  FY  1969,  procurement 
funding  in  constant  FY  2017  dollars  was  $116.8  billion.  Using  FY  2017  constant  dollars, 
real  procurement  funding  was  down  by  15.5  percent  in  FY  1970,  and  by  FY  1975  it  was 
just  over  half  what  it  had  been  in  1969.  This  decline  was  less  than  that  following  the 
Carter-Reagan  defense  buildup  but  greater  than  the  decline  of  FY  2009-FY  2012.  A 
likely  explanation  for  the  comparatively  low  cancellation  rate  during  FY  1970-FY  1975 
can  be  found  in  changes  over  the  relevant  period  in  the  composition  of  procurement 
funding.  A  large  part  of  the  increase  in  procurement  funding  during  FY  1962-FY  1969 
was  for  munitions  and  procurement  to  replace  systems  lost  in  combat,  particularly 
aircraft.  Insofar  as  procurement  of  MDAPs  for  modernization  is  concerned,  there  was 
little  or  no  boom-bust  cycle  associated  with  the  Vietnam  War.  17 

Figure  4  is  a  display  for  completion  ratios  that  parallels  that  in  Figure  3  for 
cancellations.  Of  the  162  MDAPs  for  which  we  have  a  completion  ratio,  all  but  43 
acquired  at  least  75  percent  of  their  MS  B  baseline  quantity.  The  43  MDAPs  that  did  not 
are  plotted  in  Figure  4  by  their  last  year  (the  last  year  of  production  or,  in  cases  for  which 
that  is  not  available,  the  year  of  their  final  SAR). 


16  The  model  effectively  treats  each  program  as  identical  insofar  as  cancellation  is  concerned.  There  is  a 
period  of  years  (L)  starting  at  MS  B  during  which  a  program  can  be  cancelled.  The  probability  of 
cancellation  during  the  relevant  period  is  p.  It  is  also  assumed  that  the  probability  of  cancellation  is  the 
same  in  each  year  of  L;  in  particular,  p’  =  p/L.  There  are  assumed  to  be  N  programs  at  the  start  of  each 
year,  and  any  programs  cancelled  during  the  year  are  replaced  at  the  start  of  the  next  year.  In  a  given 
year,  the  probability  that  there  will  be  k  cancellations  out  of  N  programs  is  given  by  the  binomial 
distribution  B(N,  k,  p).  The  probability  of  k  cancellations  over  s  years  is  B(sN,  k,  p'). 

17 

These  comments  are  based  on  an  unpublished  IDA  working  database  drawn  from  various  US 
government  sources.  We  are  indebted  to  Dr.  Daniel  Cuda  of  IDA  for  providing  these  data. 
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Figure  4.  Number  of  MDAPs  with  a  Completion  Ratio  of  Less  than  75  Percent, 

FY  1965-FY  2016  for  Completed  Programs 

Figure  4  does  not  convey  a  strong  impression  that  truncations  clustered  in  the 
periods  when  the  procurement  budget  was  sharply  decreasing  (FY  1986-FY  1994  and 
FY  2009-FY  2013).  The  simple  model  described  earlier  also  does  not  point  to 
clustering. 18  During  the  period  FY  1986-FY  1994,  12  MDAPs  in  the  sample  were 
concluded  before  having  procured  at  least  75  percent  of  their  MS  B  quantity.  For 
reasonable  assumptions,  the  model  finds  the  probability  of  this  observation  to  be  about  30 
percent;  that  is,  not  at  all  improbable.  The  corresponding  probability  for  the  period 
FY  2009-FY  2013  is  31  percent. 

This  is  a  surprising  and  not  entirely  plausible  result.  Just  where  the  lines  are  drawn 
on  the  periods  considered,  and  modest  changes  in  the  parameters  of  the  model,  do  not 
substantially  alter  the  finding.  Probably  of  more  importance  is  the  distinction  between 
when  a  decision  to  truncate  a  program  was  made  and  when  the  program  ended.  It  is  a 
distinct  possibility  that  in  some  cases  the  decision  to  truncate  a  program  was  taken  (say) 
two  years  before  the  program  ended.  If  so,  there  may  in  fact  be  more  clustering  around 


18 

It  was  assumed  that  70  MDAPs  were  in  production — that  is,  producing  units  that  were  fielded — and 
that  50  of  these  had  not  yet  acquired  75  percent  of  their  MS  B  baseline  quantity.  It  was  also  assumed 
that  programs  did  not  reach  75  percent  of  MS  B  baseline  quantity  until  their  eighth  year  of  production. 
Of  the  263  programs  in  the  database  that  were  not  cancelled,  43  acquired  less  than  75  percent  of  their 
MS  B  baseline  quantity,  for  an  overall  rate  of  0. 163.  The  annual  probability  over  the  seven  years  during 
which  a  program  was  liable  for  truncation  was  then  0.023  (=  0.163/7). 
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FY  1990  and  FY  2010  than  Figure  4  indicates.  On  the  evidence  at  hand,  however, 
truncations  do  not  appear  to  be  associated  with  sharp  decreases  in  procurement  funding. 

Finally,  this  discussion  raises  the  question  of  whether  the  conclusions  of  Section  C 
(on  acquisition  policy  and  process  effects)  change  if  the  two  funding  climates  (bust  and 
boom)  are  replaced  by  three  (stable,  boom,  bust).  The  question  does  not  really  arise  for 
completion  rates  of  less  than  75  percent,  which  appear  not  to  have  clusters.  Using  the 
same  approach  as  that  used  in  Section  D  but  distinguishing  three  climates  (Bust-Stable, 
Bust-Declining,  and  Boom),  no  configuration  has  a  cancellation  frequency  significantly 
different  from  that  of  the  climate  mean. 19 

E.  Cost  Growth  and  Cancellations  and  Truncations 

High  unit  cost  growth  commonly  is  thought  to  be  a  major  factor  in  MDAP 
cancellation  decisions.  This  supposition  is  plausible  because  substantial  growth  in  unit 
cost  not  only  raises  a  question  about  affordability  but  also  tends  to  call  into  question  the 
relevance  of  the  Analysis  of  Alternatives  (AoA)  that  informed  the  decision  to  acquire  the 
system:  at  some  point,  the  increase  in  unit  cost  presumably  would  tip  the  balance  in  favor 
of  another  alternative. 

The  unit  cost  metric  used  here  is  PAUC — the  sum  of  RDT&E  cost  and  procurement 
cost,  divided  by  the  number  of  units  acquired.  PAUC  growth  is  computed  by  comparing 
the  MS  B  baseline  value  of  PAUC — which  can  be  thought  of  as  a  goal  or  a  prediction — 
to  the  actual  PAUC  reported  in  the  last  SAR  for  the  program,  nonnalized  to  the  MS  B 
quantity.  (The  quantity  nonnalization  computations  are  described  in  Section  E  of 
Appendix  B;  the  nonnalization  accounts  separately  for  units  purchased  with  RDT&E 
funds  and  those  purchased  with  procurement  funds.)  Both  the  MS  B  baseline  and  the 
actual  value  of  PAUC  are  stated  in  constant  dollars.  Recall  that  “quantity  nonnalized 
PAUC  growth”  will  be  abbreviated  to  simply  “PAUC  growth.”  There  is  no  recognized 
standard  for  what  constitutes  high  PAUC  growth.  For  the  purposes  of  this  discussion,  it  is 
defined  as  PAUC  growth  against  the  MS  B  baseline  of  at  least  50  percent.20 


19  Fisher’s  Exact  Test  (FET)  provides  an  indication  that  the  DAB  may  have  had  a  significantly  higher 
cancellation  rate  in  both  Bust-Declining  and  Boom  climates  than  the  other  configurations.  There  are 
cross-climate  effects.  In  particular,  the  cancellation  rate  for  the  Bust-Declining  period  is  significantly 
higher  than  that  for  the  Bust-Stable  period.  Chi-square,  P  =  0.024. 

20 

“  The  Nunn-McCurdy  Act  in  its  current  form  defines  a  “critical”  PAUC  breach  as  one  of  50  percent  or 
more  against  the  program’s  original  baseline  (typically  the  MS  B  baseline);  a  “significant”  breach  is  one 
of  at  least  30  percent  against  the  original  baseline.  These  limits,  however,  are  for  PAUC  growth,  not 
quantity  normalized  PAUC  growth.  As  the  normalization  is  done  for  this  paper,  quantity  normalized 
PAUC  growth  is  higher  than  PAUC  growth  for  programs  that  bought  more  than  their  MS  B  baseline 
quantities  and  lower  for  those  that  bought  less. 
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A  first  interpretation  of  the  conventional  wisdom  on  unit  cost  and  cancellations  is 
that  all  programs  with  high  PAUC  growth  are  cancelled.  The  data  provided  in  the 
“Completed”  column  of  Table  8  bear  on  this  possibility.  Of  the  156  completed  MDAPs 
with  a  PAUC  growth  estimate,  43  had  PAUC  growth  of  at  least  50  percent.  The  average 
PAUC  growth  of  these  programs  was  93  percent,  and  PAUC  at  least  doubled  for  nine  of 
the  43.  Clearly,  then,  not  all  MDAPs  with  high  cost  growth  were  cancelled. 


Table  8.  Distribution  of  PAUC  Growth  for  Completed  and  Selected  Cancelled  MDAPs 


Categories 

Completed 

Cancelled 

At  least  50% 

43 

10 

Between  30%  and  50% 

19 

0 

Between  0%  and  30% 

66 

10 

Less  than  0% 

28 

5 

Total 

156 

25 

A  second  possible  interpretation  of  the  conventional  wisdom  is  that  substantially  all 
MDAPs  that  were  cancelled  had  high  unit  cost  growth.  This  possibility  is  difficult  to 
assess  because  the  final  SAR  for  an  MDAP  that  has  been  cancelled  typically  does  not 
report  the  data  required  to  estimate  PAUC  growth.21  In  some  cases,  however,  it  is 
possible  to  get  an  estimate  of  the  amount  of  cost  growth  that  occurred  prior  to 
cancellation.  This  is  done  by  going  back  to  the  most  recent  SAR  that  reported  the 
projected  costs  for  developing  the  system  and  procuring  some  quantity  of  it.  Proceeding 
in  this  way,  we  were  able  to  estimate  PAUC  growth  for  25  of  the  58  MDAPs  that  were 
cancelled. 

The  right  column  of  Table  8  presents  PAUC  growth  data  for  these  25  programs.  If 
these  data  are  taken  at  face  value,  only  10  of  the  25  cancelled  programs  showed  high  cost 
growth.  Another  10  had  PAUC  growth  of  between  0  and  30  percent,  and  five  showed 
negative  PAUC  growth.  The  conclusion  from  these  estimates  then  clearly  would  be  that 
not  all  cancelled  programs  had  high  cost  growth. 

The  PAUC  growth  estimates  in  Table  8  for  cancelled  MDAPs,  unfortunately, 
understate  the  true  PAUC  growth  in  that  they  do  not  capture  the  cost  growth  between  the 
date  of  the  SAR  used  and  the  termination  of  the  program.  In  addition,  there  tends  to  be 
some  delay  in  reporting  cost  growth  that  can  be  expected  to  occur  based  on  the  evidence 


21 

“  It  reports  the  RDT&E  funding  and  any  procurement  funding  that  will  actually  have  been  expended 
when  all  effort  on  the  program  has  ended.  Usually  these  expenditures  do  not  result  in  the  production  of 
any  fully  configured  end  items,  and  the  SAR  does  not  report  what  it  would  cost  to  complete  the 
development  program  and  procure  some  quantity  of  the  system. 
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to  date  but  which  has  not  yet  in  fact  materialized,  and  this  might  be  especially  the  case  for 
programs  that  were  cancelled.  The  data  in  Table  8,  then,  are  indicative  but  not  conclusive. 

Finally,  a  minimalist  interpretation  of  the  conventional  wisdom  is  that  the 
proportion  of  cancelled  programs  with  a  PAUC  growth  of  at  least  50  percent  is  higher 
than  it  is  for  programs  that  went  into  production.  This  is  the  case  for  the  sample  used 
here,  although  the  difference  is  not  statistically  significant.22  The  difference  for  a  PAUC 
growth  of  at  least  30  percent  also  is  not  statistically  significant  between  the  25  cancelled 
programs  and  those  that  went  into  production.23 

There  clearly  is  more  going  on  with  cancellations  than  PAUC  increases.  Combining 
the  results  on  cancellations  in  this  section  with  those  of  the  preceding  section,  it  might  be 
useful  to  consider  separately  cancellations  made  in  the  two  periods  of  sharply  declining 
procurement  funding  and  those  made  in  other  periods.  While  the  same  factors 
presumably  are  relevant,  the  relative  importance  of  those  factors  may  differ  between 
cancellations  in  the  two  bins. 

In  contrast  to  cancellation,  completion  ratios  are  a  matter  of  degree.  As  was  noted  in 
Section  B,  there  is  no  bright  line  that  separates  truncations  that  are  effectively  a  partial 
cancellation  from  those  that  are  not.  We  look  here  at  MDAPs  that  purchased  75  percent 
or  less  of  the  units  that  the  program  intended  to  purchase  at  the  time  of  MS  B.  This  line  is 
arbitrary  but  not  unreasonable. 

Table  9  provides  data  on  PAUC  growth  for  programs  with  completion  rates  of  less 
than  75  percent  versus  those  with  completion  rates  of  at  least  75  percent.  A  higher 
proportion  of  MDAPs  with  completion  rates  of  less  than  75  percent  had  PAUC  growth  of 
at  least  50  percent,  but  the  difference  is  not  statistically  significant.24  There  also  was  not  a 
significant  difference  between  the  two  groups  in  the  proportion  with  PAUC  growth  of  at 
least  30  percent.25 


22  Chi-square,  P  =  0.204. 

23  Chi-square,  P  =  1.000. 

24  Chi-square,  P  =  0.397. 

25  Chi-square,  P  =  0.888. 
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Table  9.  Distribution  of  PAUC  Growth  for  Different  Completion  Rates 


Category 

Completion  Rate  Less 
than  75% 

Completion  Rate  at 
Least  75% 

At  least  50% 

12 

28 

Between  30%  and  50% 

2 

16 

Between  0%  and  30% 

14 

51 

Less  than  0% 

9 

17 

Total 

37 

112 

There  are  sufficient  data  on  completion  rates  to  consider  a  more  ambitious  question: 
are  higher  completion  rates  associated  with  lower  PAUC  growth?  As  a  point  of  departure, 
Figure  5  is  a  scatter  for  95  MDAPs  with  a  completion  rate  of  less  than  125  percent  of  the 
MS  B  baseline  quantity. 


Figure  5.  Scatter  of  Completion  Ratios  and  PAUC  Growth  for  95  MDAPs 


This  scatter  was  constructed  as  a  “best  case”  in  support  of  a  presumption  that  as 
PAUC  growth  increases,  the  completion  rate  falls.  It  omits  eight  programs  for  which 
PAUC  growth  was  an  outlier  by  a  conventional  statistical  definition.26  In  addition,  the 
scatter  is  limited  to  programs  with  completion  rates  of  less  than  125  percent  because 
higher  completion  rates  are  positively  associated  with  quantity  nonnalized  PAUC 


26 


John  Tukey  defined  an  outlier  as  an  observation  that  is  at  least  1.5  times  the  Inter  Quartile  Range  above 
the  third  quartile  or  below  the  first. 
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growth,  although  the  correlation  is  not  statistically  significant.  This  implausible  result 
may  reflect  some  underlying  set  of  factors  more  complex  than  those  considered  here,  but 
it  seems  more  likely  that  it  is  simply  spurious.  In  particular,  it  may  be  that  the  very  high 
completion  rates  are  largely  due  to  either  (1)  instances  in  which  successive  generations  of 
a  system  are  left  on  the  same  SAR;  or  (2)  an  ill-considered  MS  B  quantity  baseline. 

The  best  fit  line  through  the  scatter  in  Figure  5  has  the  expected  negative  slope — 
completion  rates  tend  to  become  smaller  as  PAUC  growth  increases — but  the  slope  is  not 
statistically  significantly  different  from  zero.27  The  data  in  Figure  5,  however,  do  not 
support  a  clear  conclusion  about  the  association  of  completion  ratio  and  PAUC  growth 
because  they  do  not  consider  (1)  technical  performance;  (2)  the  relative  importance  of  the 
program;  (3)  the  total  size  of  the  program;  and  (4)  major  changes  in  threats,  especially 
those  that  came  with  the  end  of  the  Cold  War.  These  presumably  are  given  major  weight 
in  truncation  decisions.  Including  a  variable  for  post-Cold  War  adjustments  does  not 
substantially  alter  the  results,  however.28  Within  the  resource  limit  of  this  research, 
however,  there  was  no  prospect  of  obtaining  data  on  performance  or  relative  importance 
for  a  substantial  number  of  programs.  We  could  not,  then,  untangle  the  effects  of  PAUC 
growth  from  the  effects  of  these  factors. 

F.  Conclusions 

The  clearest  conclusion  offered  by  this  paper  is  that  cancellations  of  MDAPs  are 
concentrated  in  the  two  periods  during  which  DoD  procurement  funding  was  declining, 
often  sharply.  This  is  not  a  surprising  conclusion,  but  it  is  useful  to  know  that 
cancellation  of  major  programs  is  in  fact  one  of  the  ways  that  DoD  responded  to  large 
funding  reductions.  Truncations,  in  contrast,  were  not  clustered  in  periods  of  declining 
procurement  funding,  which  is  surprising. 

Apart  from  the  association  of  cancellations  and  decreases  in  procurement  funding, 
the  conclusions  offered  are  about  what  appears  not  to  matter — or  to  matter  much — in 
cancellation  and  truncation  decisions: 

•  Possibly  with  the  partial  exception  of  FY  1987-FY  1989  within  a  funding 
climate  there  is  no  statistical  association  between  cancellations  or  completion 
ratios  of  less  than  75  percent  and  the  acquisition  policy  and  process  prevailing 
when  a  program  passes  MS  B. 


27 

"  The  relationship  was  estimated  with  Ordinary  Least  Squares  (OLS);  the  estimated  slope  was  -10.2% 
(P  =  0.312). 

28 

“  The  adjustment  was  assumed  to  have  begun  in  FY  1991  and  continued  through  FY  2000.  The  Berlin 
Wall  fell  in  the  first  quarter  of  FY  1990  (Nov.  9,  1989).  With  the  dummy  variable  for  the  post-Cold 
War  adjustment  included,  the  estimated  slope  was  -10.1%  (P  =  0.319). 
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•  Second,  given  acquisition  policy  and  process  configuration,  there  is  no 
consistent  climate  association  of  funding  climate  with  cancellations  or 
completion  ratios  of  less  than  75  percent. 

•  Third,  while  cancelled  programs  on  average  had  higher  cost  growth  than 
programs  not  cancelled,  most  programs  with  PAUC  growth  of  more  than  50 
percent  were  not  cancelled.  Clearly,  there  is  more  behind  cancellations  than  cost 
growth.  The  evidence  does  not  permit  a  similar  conclusion  for  truncations, 
although  it  permits  those  so  inclined  to  accept  that  higher  PAUC  growth  tends  to 
result  in  lower  completion  rates. 

Is  it  reasonable  to  infer  from  these  conclusions  that  large  decreases  in  DoD  funding 
such  as  those  of  FY  1986-FY  1993  and  FY  2009-FY  2013  caused  the  cancellation  of 
MDAPs  that  otherwise  would  have  been  completed?  To  be  clear  about  this,  it  is 
necessary  to  distinguish  between  decisions  to  reduce  the  overall  DoD  acquisition 
portfolio  and  decisions  about  which  programs  to  cancel.  The  latter  involve  a  range  of 
considerations — how  well  the  programs  are  doing,  how  important  they  are,  and  the 
continued  salience  of  the  threats  to  which  they  respond,  among  others.  The  decision  on 
the  extent  to  which  cancellations  should  be  used  to  close  a  funding  gap  involves  choices 
among  bad  alternatives — cancellations,  delays  in  new  starts,  stretches  of  existing 
programs,  acceptance  of  less  costly  alternatives  in  some  cases,  and  adoption  of  very 
optimistic  costing  and  programmatic  decisions  for  both  new  starts  and — to  the  extent 
possible — ongoing  programs.  Viewed  from  this  angle,  the  root  cause  of  many 
cancellations  seems  to  be  a  mismatch  between  DoD’s  missions  and  functions,  its  force 
structure,  and  its  funding. 
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Appendix  A. 
Background 


This  appendix  provides  brief  descriptions  of  (1)  the  approach  taken  in  this  paper  to 
the  examination  of  unit  cost  growth,  (2)  each  of  the  acquisition  policy  and  process 
periods  used,  and  (3)  the  funding  climates.  The  material  is  drawn  mainly  from  Institute 
for  Defense  Analyses  (IDA)  papers  P-5 126,  P-5218,  and  P-5330  (Revised). 1 

A.  Approach 

Binning  cost  growth  in  terms  of  budget  climate  and  acquisition  policy  and  process  is 
a  departure  from  other  statistical  studies  of  cost  growth  and,  for  that  reason,  warrants  a 
brief  discussion.  Statistical  studies  on  cost  growth  in  individual  Major  Defense 
Acquisition  Programs  (MDAPs)  typically  assume  that  most  cost  growth  can  be  explained 
by  characteristics  of  the  program.  The  objective  of  these  studies  is  to  link  program 
characteristics  to  cost  growth  and  thereby  provide  a  basis  for  identifying  program 
characteristics — program  “do’s  and  do  not’s” — that  will  promote  program  success  in 
terms  of  cost,  schedule,  and  performance. 

This  paper,  in  contrast,  does  not  attempt  to  explain  why  cost  growth  occurs  but 
instead  when  it  occurs,  in  chronological  time  and  in  the  acquisition  cycle,  and  where  it 
occurs  in  terms  of  Department  of  Defense  (DoD)  processes.  The  analysis  takes  it  as  a 
point  of  departure  that  the  proximate  causes  of  Program  Acquisition  Unit  Cost  (PAUC) 
growth  are  decisions  embedded  in  programs  approved  at  Milestone  (MS)  B  (unrealistic 
cost  estimates  or  programmatic  assumptions,  for  example)  and  decisions  made  during 
program  execution  (such  as  failing  to  act  promptly  enough  on  test  results)  that  eventually 
caused  the  PAUC  growth.  With  this  assumption  in  the  background,  this  paper  and  the 
others  in  the  series  in  effect  ask  whether  factors  that  cause  (for  example)  cost  growth  tend 
to  cluster  in  certain  circumstances. 


1  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process  on 
Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5126  (Alexandria,  VA:  Institute 
for  Defense  Analyses,  September  2014);  David  L.  McNicol,  Sarah  K.  Burns,  and  Linda  Wu,  “Evidence 
on  the  Effect  of  DoD  Acquisition  Policy  and  Process  and  Funding  Climate  on  Cancellations  of  Major 
Defense  Acquisition  Programs,”  IDA  Paper  P-5218  (Alexandria,  VA:  Institute  for  Defense  Analyses, 
May  2015);  and  David  L.  McNicol  et  at,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost 
Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA: 
Institute  for  Defense  Analyses,  August  2016). 
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The  list  of  factors  that  can  be  expected  to  cause  such  clustering  is  short.  The  first  of 
these  is  acquisition  policy  and  process.  For  example,  acquisition  reforms  adopted  in  mid- 
1969  strongly  discouraged  the  use  of  fixed  price  contracts  for  development  of  MDAPs, 
and  put  in  place  a  process  for  oversight  of  compliance  with  this  and  other  acquisition 
policies.  The  second  is  funding  climate,  on  the  expectation  that  unrealistic  or  unduly 
optimistic  assumptions  are  more  likely  to  be  embodied  in  the  baselines  of  MDAPs  that 
pass  MS  B  in  bust  funding  climates. 

B.  Acquisition  Policy  and  Process  Periods 

The  President  of  the  United  States  or  the  Secretary  of  Defense,  from  DoD’s  early 
days,  occasionally  cancelled  or  initiated  major  acquisitions.  MDAPs  were  also  subject  to 
review  during  the  budget  cycle  by  the  Office  of  the  Assistant  Secretary  of  Defense 
(Comptroller)  and  the  Office  of  Management  and  Budget.  From  the  creation  of  the 
National  Security  Establishment  in  1947  through  1960,  however,  the  Office  of  the 
Secretary  of  Defense  (OSD)  had  no  institutionalized  process  for  the  oversight  of  major 
weapon  system  acquisitions. 

The  first  steps  towards  installing  an  OSD-level  acquisition  oversight  process  were 
taken  by  then  Secretary  of  Defense  Robert  McNamara  in  1964.  There  have  since  been 
many  dozens  of  changes  in  acquisition  policy  or  process  made  by  senior  OSD  officials 
and  on  the  order  of  a  dozen  major  changes  directed  by  the  Congress. 

We  organize  the  large  number  of  policy  and  process  changes  by  identifying  time 
periods  during  which  the  main  features  of  acquisition  policy  and  process  remained 
approximately  the  same.  These  are  as  follows:2 

1.  The  McNamara-Clifford  years,  FY  1964-FY  1969 

2.  The  Defense  Systems  Acquisition  Review  Council  (DSARC),  FY  1970- 
FY  1982 

3.  The  Post-Carlucci  Initiatives  DSARC,  FY  1983-FY  1989 

4.  The  Defense  Acquisition  Board  (DAB),  FY  1990-FY  1993 

5.  Acquisition  Reform  (AR),  FY  1994-FY  2000 

6.  The  DAB  Post-AR,  FY  2001  to  date 


2 

“  These  categories  are  abstracted  from  J.  Ronald  Fox,  Defense  Acquisition  Reform,  1969  to  2009 :  An 
Elusive  Goal  (Washington,  DC:  U.S.  Army  Center  of  Military  History,  2011). 
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1.  McNamara-Clifford,  1964-1969 

The  best  known  of  McNamara’s  changes  in  OSD  decision-making  processes  was 
the  creation  of  the  Office  of  Systems  Analysis  (OSA)  and  the  Planning,  Programming, 
and  Budgeting  System  (PPBS),  which  moved  decisions  on  what  major  systems  to  procure 
to  the  Secretary  of  Defense  level.  OSA  and  PPBS  so  overshadowed  development  of  an 
OSD-level  acquisition  oversight  process  that  the  latter  is  no  longer  part  of  the  collective 
memory  of  the  DoD  acquisition  community.  Nonetheless,  the  evolution  of  the  OSD-level 
acquisition  process  began  with  initiatives  taken  by  McNamara  in  the  mid-1960s. 

Provisions  for  milestone  reviews  appeared  in  1964  with  the  issuance  of  DoD 
Directive  (DoDD)  3200.9,  Initiation  of  Engineering  and  Operational  Systems 
Development ,3  This  original  version  of  the  directive  set  one  point  at  which  OSD — in 
principle,  the  Secretary  of  Defense — approval  was  required  for  an  acquisition  program  to 
proceed.  In  1965,  a  second  decision  point  was  added. 

The  two  milestones  of  the  revised  DoDD  3200.9  defined  an  acquisition  cycle  with 
three  phases.  The  first  of  these  “was  called  concept  formulation.  During  concept 
formulation  OSD  and  the  Service(s)  involved  assured  themselves  that  they  were  buying 
the  right  system  to  meet  real  needs  and  that  the  technology  was  fully  ready.”4  Concept 
formulation  typically  was  initiated  by  a  Service,  but  involved  the  Director,  Defense 
Research  and  Engineering  (DDR&E)  and  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Systems  Analysis  (OASD(SA)),  and  included  what  would  now  be  called  an  Analysis 
of  Alternatives,  led  by  OASD(SA).  It  also  apparently  included  what  would  later  be  called 
a  Mission  Element  Need  Statement  and  also  the  main  parts  of  an  Acquisition  Strategy 
and  plans  for  oversight  of  the  program  as  it  proceeded. 

Approval  to  proceed  from  the  Concept  Fonnulation  phase  authorized  the  Service 
sponsoring  the  program  to  fund  at  least  one  company  to  prepare  a  definitized  contract 
proposal.  Preparation  of  a  definitized  contract  was  the  second  phase  of  the  acquisition 
process.  The  third  phase  was  the  award  of  a  contract  for  development  and  procurement  of 
the  system.  That  is  to  say,  the  second  of  DoDD  3200. 9’s  milestones  combined  what  now 
would  be  called  MS  B  and  MS  C  authority. 

The  1965  revision  of  DoDD  3200.9  stated  that  one  of  the  purposes  of  the  Concept 
Formulation  phase  was  to  “[pjrovide  a  basis  for  a  firm  fixed  price  or  fully  structured 


3 

The  first  version  of  DoDD  3200.9  was  issued  in  1964.  A  revision  that  made  provision  for  the  Contract 
Definition  Phase  was  issued  July  1,  1965.  See  Thomas  K.  Glennan,  Jr.,  “Policies  for  Military  Research 
and  Development,”  RAND  Paper  P-3253  (Santa  Monica  CA:  The  RAND  Corporation,  1966),  12. 

4  William  D.  O’Neil  and  Gene  H.  Porter,  “What  to  Buy?  The  Role  of  Director  of  Defense  Research  and 
Engineering  (DDR&E) — Lessons  from  the  1970s,”  IDA  Paper  P-4675  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  January  2011),  30. 
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incentive  contract  for  Engineering  Development.”5  In  addition,  McNamara  directed  the 
use  of  Total  Package  Procurement  (TPP)  when  it  was  judged  to  be  practicable  and,  when 
not,  a  Fixed  Price  Incentive  Fee  (FPIF)  or  Cost  Plus  Incentive  Fee  (CPIF)  contract.  By 
1966,  McNamara  had  concluded  that  TPP  contracts  were  in  fact  not  a  practicable  way  to 
acquire  most  major  weapon  systems,6  although  acquisition  policy  apparently  still  had  a 
tilt  towards  fixed  price  contracts,  even  for  development. 

The  OSD-level  milestone  reviews  were  structured  around  what  was  then  called  the 
Development  Concept  Paper  (DCP).  The  requirement  for  a  DCP  was  instituted  in  1965 
by  the  DDR&E,7  and  in  1968,  it  was  required  in  order  to  initiate  any  major  development 
project.  DDR&E  coordinated  initial  DCPs  with  concerned  OSD  offices  (and  probably  the 
Joint  Staff  and  other  Services)  and  with  what  now  would  be  called  the  Milestone 
Decision  Authority  (MDA)  for  the  initial  DCP.8  Once  approved  by  DDR&E,  the 
proposed  new  start  went  to  the  Secretary  of  Defense,  although  the  sources  consulted  do 
not  indicate  whether  it  went  as  a  separate  action  or  as  part  of  the  Service’s  budget 
submission.  It  is  also  not  clear  which  OSD  official  was  the  MDA  for  the  second 
milestone. 

2,  The  1969  Packard  Reforms 

The  start  of  the  second  acquisition  period  is  marked  by  reforms  initiated  by  then 
Deputy  Secretary  of  Defense  David  Packard  in  mid-1969.  The  elements  of  the  Packard 
reforms  fall  into  three  main  categories.  First,  Packard  clarified  and  formalized  the  OSD- 
level  acquisition  oversight  process  that  had  emerged  under  McNamara  and  Clifford.  This 
was  achieved  through  a  combination  of  steps: 

•  The  DSARC  was  established  to  advise  the  MDA  at  each  milestone. 

•  DDR&E  was  designated  as  the  MDA  at  MS  I  and  MS  II;  the  Assistant  Secretary 
of  Defense  for  Installations  and  Logistics  was  the  MDA  for  MS  III. 

•  The  Development  Concept  Paper  was  renamed  the  Decision  Coordinating  Paper 
(retaining  the  acronym). 


5  DoDD  3200.9,  Initiation  of  Engineering  and  Operational  Systems  Development,  July  1,  1965,  V.B(l). 

6  Gordon  Adams,  Paul  Murphy,  and  William  Grey  Rosenau,  Controlling  Weapons  Costs:  Can  Pentagon 
Reforms  Work?  (New  York:  Council  on  Economic  Priorities,  1983),  19-20.  A  TPP  contract  is  one  that 
covers  EMD,  at  least  a  significant  portion  of  procurement,  and  at  least  part  of  the  support  of  the  system 
(for  example,  depot  maintenance).  Fox  also  notes  that  McNamara  moved  to  consolidate  acquisition 
functions  in  defense  agencies — e.g.,  the  agency  that  became  the  Defense  Logistics  Agency — and 
promoted  the  use  by  program  managers  of  particular  management  tools  such  as  PERT  and  earned  value. 

7  Fox,  Defense  Acquisition  Reform,  1969  to  2009,  24-47,  provides  a  sketch  of  how  the  process  evolved 
and  worked  during  the  1960s. 

8 

See  C.  W.  Borklund,  The  Department  of  Defense  (New  York:  Frederick  A.  Praeger,  1969),  83. 
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•  Acquisition  policy  and  the  process  of  OSD-level  milestone  and  program  reviews 
were  laid  out  in  a  new  directive  (DoDD  5000.1)  and  a  new  top-level  process 
instruction  (DoDI  5000.2). 9 

As  each  of  these  steps  had  antecedents  in  the  McNamara-Clifford  process,  it  seems 
reasonable  to  say  that  they  were  evolutionary  rather  than  revolutionary. 10 

Second,  Packard  made  changes  to  policy  on  contract  types.  Picking  up  on  this  topic 
where  McNamara  had  left  off,  he  ruled  out  the  use  of  TPP  and  discouraged  the  use  of 
FPIF  for  development  contracts  in  favor  of  CPIF.  (Cost  Plus  Award  Fee  may  not  have 
been  included  in  the  contracting  play  book  yet).  As  a  general  matter,  Packard’s  policy 
was  to  match  contract  terms  to  the  riskiness  of  the  acquisition. 

Third,  and  finally,  Packard  redefined  the  milestones  so  as  to  separate  the  decision  to 
allow  the  program  to  enter  Engineering  and  Manufacturing  Development  (EMD)  (now 
MS  B)  from  the  decision  to  enter  the  Production  phase  (now  MS  C)  and  required 
OSD-level  approval  of  each  decision.  He  also  collapsed  DoDD  3200. 9’s  Contract 
Definition  phase  into  the  new  and  broader  Validation  phase,  which  has  at  various  times 
since  been  called  Demonstration  and  Validation,  Program  Development  and  Risk 
Reduction  and,  currently,  Technology  Maturation  and  Risk  Reduction.  MS  I  (now  MS  A) 
authorized  entry  into  this  phase.  These  changes  were  more  revolutionary  than 
evolutionary. 1 1 

Packard’s  reforms  were  at  the  time  explained  as  a  return  to  Components  of  the 
authority  to  manage  major  acquisition  programs.  McNamara  had  an  active  role  making 
decisions  about  some  major  programs,  and  against  this  background  Packard’s  reforms 
were  a  return  to  Components  of  management  responsibility.  Under  the  new  acquisition 
directives,  the  Secretary  of  Defense,  while  retaining  full  legal  authority  over  acquisition 
programs,  would  act  through  the  established  acquisition  process  except  in  extraordinary 
circumstances.  Decisions  at  the  DSARC  level  were  advisory  to  the  Secretary  and  Deputy 
Secretary  of  Defense  but,  apart  from  exceptional  cases,  they  probably  reached  that  level 
by  way  of  the  Service’s  proposed  budgets  (and  the  Comptroller  was  the  backstop 
enforcer  of  the  requirement  for  milestone  approval  before  a  program  could  advance  to  the 
next  stage).  This  implied  less  OSD-level  control  over  some  major  acquisitions — those  in 


9  DoDD  5000.1,  The  Defense  Acquisition  System,  May  12,  2003;  and  DoDI  5000.2,  Operation  of  the 
Defense  Acquisition  System,  May  12,  2003. 

10  Fox,  Defense  Acquisition  Reform,  1969  to  2009,  57,  provides  a  useful  schematic  comparison  of  the 
DoDD  3200.9  milestones  and  those  of  Packard’s  DoDD  5000.1/DoDI  5000.2. 

1 1  DoDI  5000.2,  issued  October  23,  2000,  formally  established  MSs  A,  B,  and  C  (in  place  of  MSs  I,  II, 
and  III)  as  the  main  decision  points  for  an  MDAP.  The  definitions  are  such  that  MS  B  is  placed  several 
months  earlier  in  the  process  than  MS  II. 
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which  McNamara  had  been  actively  involved.  The  DSARC,  however,  had  a  greater 
substantive  scope  than  the  1960s  process  for  the  more  typical  program  and  was  more 
tightly  organized.  For  the  large  majority  of  major  acquisition  programs,  then,  the  new 
DSARC  process  probably  was  more  effective. 12 

The  process  instituted  by  Packard  in  mid- 1969  had  all  of  the  basic  features  that  are 
still  central  to  OSD-level  oversight  of  MDAPs — three  milestones,  with  definitions  similar 
to  those  used  now;  formal  milestone  reviews;  a  body  to  advise  the  MDA;  and  a 
document(s)  that  described  the  basic  features  of  the  program,  provided  a  vehicle  for  staff 
inputs,  and  set  down  the  cost,  schedule,  and  performance  goals  that  the  program  was  to 
meet.13  This  is  an  important  point  because  it  implies  that  the  effects  of  Packard’s  reforms 
would  endure,  so  long  as  the  process  was  not  altered  in  a  fundamental  way. 

3.  The  More  Recent  Periods 

The  vehicle  for  the  transition  from  the  first  phase  of  the  DSARC  (FY  1970- 
FY  1982)  to  the  second  (FY  1983-FY  1989)  was  the  Carlucci  Initiatives,  named  after 
then  Deputy  Secretary  of  Defense  Frank  Carlucci.  These  were  developed  during  CY  1981 
and  implemented  during  FY  1982-FY  1983.  The  Carlucci  Initiatives  did  not  involve  any 
major  changes  in  the  DSARC  process  or  in  the  policies  Packard  had  established;  in  these 
tenns,  the  Carlucci  Initiatives  were  more  reaffirmation  than  change.  The  Initiatives, 
however,  included  measures  designed  to  coordinate  decisions  on  MDAPs  made  in  the 
PPBS  and  those  made  in  the  DSARC  process.  Important  statutory  changes  were  also 
made  during  1982  and  1983. 


12 

“  Clark  A.  Murdock,  Defense  Policy  Formation  (Albany,  NY :  State  University  of  New  York  Press, 

1974),  155-179,  disagrees  with  this  judgment.  Murdock  is  primarily  concerned  with  Systems  Analysis 
and  resource  allocation,  but  also  comments  specifically  on  the  acquisition  process.  In  particular,  he 
notes  that  the  new  Decision  Coordinating  Paper  did  not  provide  “any  mechanism  for  ongoing 
managerial  control.”  This  is  accurate  in  that  the  Packard  reforms  placed  management  of  the  programs  in 
the  hands  of  the  Services.  It  is  incomplete  in  that  the  Services  were  responsible  for  staying  within  what 
would  later  be  called  the  Acquisition  Program  Baseline,  and  the  MDA  was  enjoined  to  act  in  cases  in 
which  they  did  not. 

13 

In  January  1972,  the  DSARC  process  was  expanded  to  include  an  independent  cost  estimate  at  MS  II 
and  MS  III  provided  by  the  Cost  Analysis  Improvement  Group  (CAIG),  newly  established  by  a 
memorandum  signed  by  Melvin  Laird  on  January  25,  1972.  A  December  7,  1971  memorandum  signed 
by  David  Packard  directed  the  Military  Departments  to  begin  using  “independent  parametric  cost 
analysis.”  See  Donald  Srull,  ed..  The  Cost  Analysis  Improvement  Group:  A  History’  (McLean,  VA: 
Logistics  Management  Institute  (now  LMI),  1998),  47-48.  Since  the  implementation  of  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  (WSARA),  the  independent  cost  estimates  have  been 
provided  by  the  Cost  Assessment  Deputate  of  the  Office  of  Cost  Assessment  and  Program  Evaluation 
(CAPE). 
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The  DSARC  was  followed  by  the  DAB.14  While  the  DAB  itself  bears  a  strong 
family  resemblance  to  the  DSARC,  the  statute  that  created  it  also  created  the  position  that 
is  now  called  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(USD(AT&L))  and  Service  Acquisition  Executives,  who  reported  to  the  USD(AT&L); 
designated  USD(AT&L)  as  the  MDA  for  most  MDAPs;  and  removed  the  Service  chiefs 
from  the  acquisition  chain  of  command.  In  addition,  the  statute  created  the  position  of 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  (VCJCS)  and  prompted  a  new  requirements 
process  centered  on  the  VCJCS. 

AR  (FY  1994-FY  2000)  was  to  a  large  extent  intended  to  put  acquisition  of  major 
weapon  systems  on  a  more  commercial  basis  and  make  it  easier  for  firms  outside  the 
defense  sector  to  sell  to  DoD.  Contracts  were  structured  so  that  defense  contractors 
assumed  more  responsibility  for  system  performance;  correspondingly,  DoD’s  role  in 
contract  management  decreased.  OSD  oversight  of  MDAPs  also  was  relaxed  somewhat. 
Substantial  cuts  in  acquisition  staffs  at  both  the  OSD  level  and  Service  Headquarters 
level  were  made,  and  senior  decision  makers  took  a  more  pennissive  attitude  towards 
cost  growth. 

The  post-AR  DAB  period  was  marked  by  the  arrival  of  a  new  administration  in 
January  2001.  This  period  saw  no  overt  rejection  of  AR,  but  most  of  its  initiatives  were 
no  longer  pursued.  The  new  administration  seemed  implicitly  to  favor  a  return  to  the 
status  quo  ante  in  OSD-level  oversight  of  acquisition  programs. 

In  2006,  the  Congress  strengthened  the  Nunn-McCurdy  Act,  which  had  been  passed 
in  1982. 15  In  2009,  the  Congress  passed  the  Weapon  Systems  Acquisition  Reform  Act 
(WSARA),  which  made  several  changes  that  may  prove  to  be  consequential. 

C.  Funding  Climates 

It  is  important  to  realize  that  funds  appropriated  for  procurement  are  a  lagging 
indicator  of  a  change  in  budget  climate.  Funding  and  programmatic  decisions  embedded 
in  the  MS  B  baselines  of  MDAPs  are  made  at  least  one  or  two  years  before  the 
President’s  Budgets  in  question  are  submitted  to  the  Congress.  Consequently,  those 
decisions  necessarily  reflect  expectations  held  by  decision  makers  about  the  future  DoD 


14  There  is  some  uncertainty  about  when  the  Post-Carlucci  Reforms  DSARC  should  end  and  the  DAB 
period  should  begin.  The  relevant  statutes  were  passed  in  December  1985  (first  quarter  FY  1986),  and 
the  DAB  began  functioning  under  that  name  in  late  FY  1987  or  early  FY  1988;  however,  DoD  did  not 
implement  the  full  set  of  reforms  required  by  statute  until  1990.  We  have  for  that  reason  set  the  line  at 
1990. 

15  The  Nunn-McCurdy  Act  imposes  reporting  and  other  requirements  that  senior  DoD  officials  must  meet 
if  an  MDAP  breaches  certain  unit  cost  thresholds. 
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budget  climate,  and  the  breakpoints  between  different  budget  climates  should  mark  the 
points  at  which  there  were  major  shifts  in  expectations. 

It  is  also  important  to  recognize  the  distinction  between  the  development  of  the  DoD 
budget  request  for  acquisition  and  the  amount  the  Congress  eventually  appropriates.  The 
Components’  acquisition  programs  are  built  subject  to  funding  constraints  specified  by 
senior  Component  officials  (in  particular,  the  Service  Secretaries)  based  on  fiscal 
guidance  they  are  given  by  the  Secretary  of  Defense.  It  is  reasonable  to  assume  that 
expectations  about  future  funding  levels  are  fonned  as  much  by  “this  year’s”  fiscal 
guidance  as  by  “last  year’s”  congressional  appropriation. 

We  used  four  events  to  identify  the  breakpoints  between  funding  climates:  (1)  the 
invasion  of  Afghanistan  by  the  USSR  in  late  December  1979;  (2)  passage  of  the  Gramm- 
Rudman-Hollings  Act  in  December  1985;  (3)  the  terrorist  attack  on  the  United  States  on 
September  11,  2001;  and  (4)  the  start  of  the  withdrawal  of  US  forces  from  Iraq  in  late 
FY  2009.  Senior  decision  makers  could  reasonably  expect  each  of  the  events  identified  to 
signal  major  and  sustained  changes  in  the  defense  funding  climate,  which  in  fact  they  did. 

1.  The  First  Bust  Climate 

DoD  procurement  funding  increased  markedly  during  the  first  three  years  of  the 
Kennedy  administration  (CY  1961-CY  1963)  and  remained  relatively  high  through  the 
1960s.  A  large  part  of  the  increase  in  procurement  funding  during  FY  1964-FY  1969, 
however,  was  for  munitions  and  procurement  to  replace  systems  lost  in  combat, 
particularly  aircraft.  Insofar  as  modernization  of  weapon  systems  was  concerned,  there 
was  little  or  no  boom  associated  with  the  Vietnam  War. 16 

Gerald  Ford  was  sworn  in  as  president  on  August  9,  1974,  about  six  weeks  into 
FY  1975.  The  FY  1976  budget,  preparation  of  which  was  far  advanced  when  Ford 
became  president,  increased  procurement  by  13.5  percent.  The  amount  appropriated  for 
procurement  in  FY  1977  increased  by  an  additional  30  percent,  and  the  amounts 
appropriated  for  procurement  during  the  next  two  years  were  only  modestly  lower. 

There  are  three  main  reasons  for  concluding  that  the  FY  1976  and  FY  1977 
increases  did  not  cause  a  major  shift  in  expectations  about  defense  spending.  First,  they 
basically  reversed  sharp  declines  in  procurement  funding  during  the  years  1970-1975, 
roughly  the  period  of  the  “Peace  Dividend”  that  had  accompanied  the  end  of  the  War  in 
Vietnam. 


16  These  comments  are  based  on  an  unpublished  IDA  working  database  drawn  from  various  US 
government  sources.  We  are  indebted  to  Dr.  Daniel  Cuda  for  providing  these  data. 
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Second,  there  was  no  political  consensus  in  favor  of  a  sustained  substantial  increase 
in  funding  for  DoD  procurement.  As  evidence  of  this  point,  during  the  1976  campaign, 
President  Carter  “pledge[d]  to  reduce  the  defense  budget  by  $7  billion  and  submit  a 
balanced  budget  in  fiscal  year  1981.” 17 

Third,  while  Ford  opposed  cuts  to  defense  spending,  the  material  consulted  for  this 
paper  (Ford’s  three  State  of  the  Union  addresses  and  campaign  material)  did  not  indicate 
that  he  attempted  to  establish  a  consensus  favoring  large  increases  in  DoD  spending.  It 
also  is  far  from  clear  that  he  was  in  a  position  to  do  so  during  most  of  his  presidency.  The 
Democrats  controlled  both  houses  of  the  Congress.  In  his  inaugural  address  to  the 
Congress,  Ford  said:  “I  am  your  man,  for  it  was  your  carefully  weighed  confirmation  that 
changed  my  occupation.”  That  is,  his  legitimacy  as  president  was  derived  from  the 
Congress  rather  than  from  election  by  the  people. 

The  Carter  administration  followed  through  on  his  campaign  commitment  by 
reducing  the  FY  1978  DoD  budget  request  left  by  the  Ford  administration  by  about  $3 
billion.  Moreover,  on  August  17,  1978,  President  Carter  vetoed  the  1979  Defense 
Authorization  Act.  The  veto  message  implies  that  the  Congress  had  put  a  $2  billion 
nuclear  powered  carrier  into  the  budget  Carter  had  submitted  from  the  Ford 
administration  and  cut  some  other  procurement  programs,  some  readiness  funding,  and 
some  RDT&E  funding.  Carter  wanted  the  carrier  removed  from  the  budget  and  the  other 
funding  he  had  requested  restored.  The  veto  was  sustained.  As  these  facts  suggest  would 
be  the  case,  DoD  Budget  Authority  (BA)  was  lower  for  FY  1978  and  FY  1979  than  it  was 
for  FY  1977,  and  FY  1980  was  only  slightly  larger. 

2.  Start  of  the  First  Boom  Climate 

The  Soviet  invasion  of  Afghanistan  in  late  December  1979  (three  months  into 
FY  1980)  prompted  a  change  in  the  Carter  administration’s  policy  on  defense  spending. 
One  source  states  that  “only  a  month  after  the  Soviet  invasion  of  Afghanistan,  Carter 
called  for  increases  in  defense  spending  of  4.6  percent  per  year,  every  year  over  five 
years.”18  Carter’s  State  of  the  Union  address  delivered  on  January  21,  1980,  listed  as  one 
of  his  few  new  proposals  “initiatives  implementing  my  response  to  the  Soviet  invasion  of 
Afghanistan.”  The  “Defense  Spending”  portion  of  the  State  of  the  Union  address  stated 
that  requested  FY  1981  funding  authority  was  a  more  than  5  percent  real  increase  over 
the  funding  requested  for  FY  1980.  It  goes  on  to  characterize  this  as  “a  growth  rate  for 


17  >> 

Frank  L.  Jones,  A  "Hollow  Army  ”  Reappraised:  President  Carter,  Defense  Budgets,  and  the  Politics  of 
Military  Readiness  (Carlisle,  PA:  Strategic  Studies  Institute,  U.S.  Army  War  College,  2012),  16. 

18 

Andrew  Krepinevich,  “What  Would  Jimmy  Carter  Do?,”  Defense  One,  March  11,  2014, 

http://www.defenseone.com/ideas/2014/03/what-would-jimmy-carter-do/80335/7oreUsearch_ 

what%20would%20jimmy%20carter%20do. 
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defense  that  we  can  sustain  over  the  long  haul.”  In  fact,  the  real  increase  in  enacted  DoD 
BA  for  FY  1981  was  12  percent  for  DoD  as  a  whole  and  about  25  percent  for 
procurement. 

President  Reagan,  who  took  office  in  January  1981,  requested  increases  in  DoD 
funding  that  went  well  beyond  what  the  Carter  administration  had  planned.  By  FY  1985, 
DoD  funding  for  procurement  was  more  than  70  percent  above  what  it  had  been  in 
FY  1981. 

3.  The  Second  Bust  Period 

Congressional  appropriations  for  defense  procurement  peaked  in  FY  1985.  The 
Congress  did  not  enact  the  increase  in  DoD  funding  for  FY  1986  requested  by  the 
administration,  and  in  December  1985  passed  the  Gramm-Rudman-Hollings  (GRH)  Act, 
which  provided  for  sequestration  to  satisfy  budget  targets. 

None  of  this  was  known,  however,  when  the  FY  1986  DoD  budget  request  was  sent 
to  the  Congress  in  late  January  or  early  February  1985,  and  the  defense  buildup  was  in 
full  swing  during  FY  1983  through  FY  1985  when  the  FY  1986  request  was  being 
developed.  For  that  reason,  this  paper  takes  FY  1986  as  the  last  year  of  the  Carter-Reagan 
boom  climate. 

The  Congress  cut  DoD  funding  for  FY  1987  about  3  percent  below  the  FY  1986 
level.  Cuts  in  defense  spending  continued  through  the  end  of  the  Reagan  administration. 
Procurement  funding  for  FY  1987  was  down  about  16  percent  from  its  FY  1986  level  and 
dropped  modestly  further  over  the  next  three  fiscal  years.  Additional  large  drops  came 
after  the  fall  of  the  Berlin  Wall  on  November  9,  1989  (first  quarter  FY  1990)  and  the 
collapse  of  the  Soviet  Union  on  December  25,  1991  (first  quarter  FY  1992).  By  the  mid- 
1990s,  DoD  procurement  funding  had  returned  to  about  the  post-Vietnam  War  levels  of 
the  mid-1970s  and  was  little  more  than  one-third  of  its  peak  (FY  1985)  level.  Altogether, 
the  second  bust  phase  continued — for  a  total  of  16  years — through  FY  2002. 

4.  The  Second  Boom  Period 

The  defense  spending  boom  that  followed  was  prompted  by  the  A1  Qaeda  attacks  of 
September  11,  2001  (late  FY  2001)  and  the  subsequent  wars  in  Afghanistan  and  Iraq.  The 
question  is  where  the  start  of  the  boom  climate  should  be  placed — at  FY  2001,  FY  2002, 
or  FY  2003? 

George  W.  Bush  was  inaugurated  on  January  20,  2001.  It  is  not  clear  that  the 
outgoing  Clinton  administration  submitted  its  final  (FY  2002)  budget  to  the  Congress. 
The  Bush  administration  apparently  submitted  a  pro  fonna  budget  (A  Blueprint  for  New 
Beginnings)  on  February  29,  2001,  and  a  complete  FY  2002  request  proposal  on  April  9. 
The  Bush  administration  also  requested  a  $5.2  billion  increase  in  the  FY  2001 
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appropriation.  According  to  the  budget  news  release,  this  was  primarily  for  Military 
Personnel  (MILPERS)  and  Operations  and  Maintenance  (O&M).  As  a  candidate,  Bush 
had  not  committed  to  increasing  procurement,  and  his  proposed  budget  for  FY  2002  did 
not  do  so.  In  fact,  procurement  was  actually  $1.4  billion  lower  than  the  FY  2002  column 
of  the  FY  2001  Clinton  Future  Years  Defense  Plan. 

Post  9/11,  the  budget  climate  was  radically  different.  On  December  28,  2001  (end  of 
the  first  quarter  of  FY  2002),  the  Congress  adopted  an  emergency  supplemental  for 
FY  2001,  even  though  it  had  ended  three  months  earlier,  and  on  January  10,  2002 
adopted  an  emergency  supplemental  for  FY  2002.  The  spending  plans  that  went  into  the 
proposed  FY  2002  budget  could  not  have  been  influenced  by  these  events. 

Planning  for  the  FY  2004  DoD  budget  certainly  took  place  in  a  boom  climate.  That 
is  less  clearly  true  of  the  FY  2003  budget.  The  FY  2003  DoD  budget  submitted  February 
4,  2002  was  about  10  percent  above  the  FY  2002  budget  request  and,  more  important  for 
present  purposes,  provided  for  about  a  10  percent  increase  in  procurement  funding. 
Barely  four  months  elapsed  between  9/11  and  the  submission  of  the  FY  2003  DoD 
budget.  On  the  supposition  that  major  changes  in  acquisition  programs  could  have  been 
made  within  that  amount  of  time,  this  paper  places  the  start  of  the  post-9/1 1  boom  climate 
at  FY  2003. 

DoD  funding  for  procurement  increased  in  FY  2003  by  about  20  percent  over  the 
preceding  year  and  continued  to  increase  through  FY  2008.  FY  2009  probably  should  be 
counted  as  the  last  year  of  this  second  boom  period  because  the  Great  Recession  began  in 
its  first  quarter  and  the  withdrawal  of  US  troops  from  Iraq  began  in  July  of  2009. 
Expectations  of  a  decline  in  defense  spending  developed  at  that  time,  and  were  solidified 
by  passage  of  the  Budget  Control  Act  of  2011,  which  was  signed  into  law  in  August 
2011. 
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Appendix  B 
The  Data 


A  compact  disc  (CD)  in  a  pocket  on  the  inside  back  cover  of  this  paper  contains  the 
database  used  in  this  research.  The  CD  also  contains  several  other  files  that  help  to 
document  or  explain  the  data.  This  appendix  provides  background  infonnation  that 
facilitates  access  to  and  use  of  the  files  on  the  CD. 

A.  Cost  Growth  Metric  and  Ground  Rules 

The  principal  cost  growth  metric  used  in  this  paper  is  quantity  normalized  Program 
Acquisition  Unit  Cost  (PAUC)  growth  in  program  base  year  dollars.  In  most  instances, 
the  PAUC  growth  figure  used  is  measured  from  the  MS  B  baseline.  PAUC  includes 
Research,  Development,  Test,  and  Evaluation  (RDT&E)  funding  as  well  as  procurement 
funding. 

Each  of  the  programs  in  the  database  with  a  PAUC  growth  estimate  completed 
Engineering  and  Manufacturing  Development  (EMD),  went  into  production,  and  fielded 
at  least  some  units  to  operating  forces.  We  follow  the  convention  of  not  including  in  the 
database  any  MDAP  that  was  not  at  least  five  years  beyond  EMD,  so  that  cost  growth 
would  have  time  to  appear.  The  most  recent  Selected  Acquisition  Reports  (SARs) 
available  for  this  paper  were  those  for  December  2015,  so  the  most  recent  programs 
included  are  those  that  passed  MS  B  in  FY  2010.  The  database  contains  an  estimate  of 
PAUC  growth  for  184  of  the  MDAPs  that  entered  EMD  during  FY  1965-FY  2010. 

The  estimates  mainly  are  drawn  from  the  database  developed  for  IDA  Paper  P-5330 
(Revised),  which  in  turn  evolved  from  the  database  for  IDA  Paper  P-5126.1  The  cost 
growth  observations  for  FY  1965-FY  1969,  however,  and  a  few  of  the  observations  for 
FY  1965-FY  1989,  are  drawn  from  other  studies,  as  is  discussed  below. 

Figure  B-l  is  a  histogram  of  the  cost  growth  estimates  in  the  database  used  for  this 
paper.  The  paper  used  the  162  completed  MDAPs  that  passed  MS  B  during  the  period 


1  David  L.  McNicol  et  at,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of 
Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  August  2016);  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD 
Acquisition  Policy  and  Process  on  Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper 
P-5126  (Alexandria,  VA:  Institute  for  Defense  Analyses,  September  2014). 
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FY  1965-FY  2009  for  which  both  an  Average  Procurement  Unit  Cost  (APUC)  and  a 
PAUC  estimate  were  available. 


Figure  B-1.  Histogram  of  Quantity  Normalized  PAUC  Growth  from  the  MS  B  Baseline  for 

Completed  MDAPs 


B.  Business  Rules 

Almost  all  of  the  data  used  in  this  research  were  taken  directly  or  indirectly  from 
SARs.  SARs  filed  in  FY  1997  and  subsequent  years  are  available  through  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  system.  Many  SARs  filed 
before  FY  1997  are  available  on  an  Office  of  the  Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics)  (OUSD(AT&L))  SIPRNet  site.  These  two 
sources  provided  SARs  under  about  345  distinct  labels. 

Not  all  of  these  distinct  labels  are  distinct  programs.  Three  steps  are  needed  to  get 
from  the  list  of  distinct  SAR  labels  to  a  list  of  MDAPs: 

1.  During  the  1970s,  each  Component  involved  in  a  joint  program  sometimes  filed 
a  SAR.  These  SARs  reported  the  same  program  data.  The  database  used  in  this 
research  includes  only  the  data  reported  (for  the  entire  program)  in  the  SAR 
filed  by  the  lead  Component. 

2.  The  program  name  used  on  the  SAR  often  changes  over  the  acquisition  cycle  for 
a  given  program.  For  example,  the  OH-58D  Kiowa  Warrior  was  first  reported  as 
the  Army  Helicopter  Improvement  Program  (AHIP).  In  most  cases  the  database 
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uses  the  name  under  which  the  last  (or,  for  ongoing  programs,  most  recent)  SAR 
was  filed. 

3.  Multiple  MDAPs  that  have  passed  MS  B  are  sometimes  combined  into  a  single 
MDAP.  Conversely,  a  single  MDAP  that  has  passed  MS  B  is  sometimes  split 
into  two  or  more  separate  MDAPs.  If  the  data  permitted  (and  they  often  did  not), 
our  rule  was  to  maintain  the  program(s)  as  they  had  been  defined  at  MS  B. 

For  the  reasons  noted  above,  the  database  does  not  include  any  MDAPs  that  passed 
MS  B  after  FY  2010.  In  addition,  the  following  were  excluded  from  the  main  database: 

•  Major  Automated  Infonnation  Systems  (MAIS), 

•  Chemical  Demilitarization  Programs, 

•  Ballistic  Missile  Defense  programs  managed  by  the  Ballistic  Missile  Defense 
Agency  and  its  predecessors, 

•  Programs  that  filed  a  SAR  but  were  never  designated  as  an  MDAP,  and 

•  Programs  cancelled  before  they  passed  MS  B  or  before  they  were  designated  as 
an  MDAP. 

These  exclusions  were  indicated  by  the  purpose  of  the  analysis,  which  is  to  gauge  the 
effect  of  different  OSD-level  acquisition  regimes  and  funding  climates  on  MDAP 
outcomes.  The  database  then  should  include  only  programs  subject  to  OSD-level 
acquisition  policy  and  process.  To  at  least  a  significant  extent,  the  excluded  programs 
differed  from  the  MDAP  norm.  The  exclusions  therefore  resulted  in  a  main  database  that 
includes  316  MDAPs  that  entered  development  during  the  period  FY  1965-FY  2010. 

Most  of  the  MDAPs  in  the  database  passed  MS  B  at  the  OSD  level.  Some,  however, 
entered  at  MS  III/C,  obtained  both  MS  B  and  Low  Rate  Initial  Production  authority  in  a 
single  OSD-level  review,  or  passed  MS  B  at  the  Service  level  and  later  became 
Acquisition  Category  (AC AT)  I  programs.  These  cases  are  noted  in  the  database  for 
programs  that  became  MDAPs  in  FY  1989  or  later,  but  not  reliably  noted  for  programs 
begun  earlier. 

Finally,  it  proved  to  be  necessary  to  adopt  a  clear  criterion  for  program  cancellation. 
In  the  database,  a  program  is  classified  as  cancelled  if: 

•  The  program  did  not  result  in  production  of  any  fully  configured  end  items,  or 

•  Any  fully  configured  end  items  produced  were  used  only  for  testing  and 
development. 

Application  of  this  definition  was  not  clear-cut  for  six  programs  that  passed  MS  B  at  the 
Service  level,  later  filed  SARs,  and  subsequently  were  cancelled.  We  retained  on  the  list 
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of  cancelled  programs  the  five  that  had  been  designated  as  an  ACAT  I  program  and 
excluded  the  one  that  had  not.2 

Two  other  programs  were  counted  as  cancelled,  although  they  did  not  exactly 
satisfy  the  criteria  stated.  The  C-27J  was  included  on  the  list  of  cancelled  programs 
because  the  21  C-27Js  produced  were  placed  directly  in  long-term  storage  and  later 
transferred  to  Special  Operations  Command  and  the  US  Coast  Guard.  Roland  was 
included,  although  the  system  was  produced  in  the  United  States  in  limited  quantities  and 
issued  to  a  single  National  Guard  battalion,  which  falls  into  a  gray  area  between  issue  of 
the  system  to  active  duty  units  and  its  use  only  for  development,  experiment,  and 
training. 

The  file  “Program  Notes”  (provided  on  the  included  CD)  provides  some  information 
bearing  on  each  of  the  58  cancellations  we  identified.  We  found  12  additional  programs 
that  filed  one  or  more  SARs  during  FY  1959-FY  2009  and  were  cancelled.  These  12 
were  not  included  on  the  list  of  cancelled  programs  because  they  were  either  cancelled 
before  passing  MS  B,  were  never  designated  an  ACAT  I  program,  or  were  cancelled  after 
they  fell  below  the  ACAT  I  level.  They  appear  as  numbers  59-70  in  “Program  Notes.” 

C.  Coverage 

As  was  noted  above,  the  database  includes  58  MDAPs  that  were  cancelled  (as  an 
ACAT  I  program)  after  passing  MS  B  and  includes  254  programs  that  went  into 
production.  We  have  APUC  and  PAUC  estimates  for  185  of  the  MDAPs  that  went  into 
production,  of  which  32  were  still  underway  as  of  the  December  2015  SARs.  Table  B-l 
reports  the  relevant  data  broken  down  by  the  nine  time  periods  used  in  the  statistical 
analysis.  Overall,  the  database  reports  a  cost  growth  estimate  for  about  70  percent  of  the 
MDAPs  that  went  into  production. 


2  AN/WQR-Advanced  Deployable  System,  AQM-127A  Supersonic  Low  Altitude  Target,  Advanced  Seal 
Delivery  System,  ASM-135A  Air-Launched  Anti-Satellite  System,  and  Land  Warrior  were  retained  on 
the  list  of  cancelled  programs.  Extended  Range  Munition  was  cancelled  before  it  was  designated  an 
ACAT  I  program. 
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Table  B-1.  MDAPs  in  the  Database  Not  Cancelled,  with  an  APUC  and  a  PAUC  Estimate,  by 

Bust/Boom  Time  Periods 


Period  (FY) 

Went  into 
Production 

No.  with 
APUC  & 
PAUC 

Percent 
with  APUC 
&  PAUC 

1965-1969 

21 

16 

76% 

1970-1980 

62 

49 

79% 

1987-1989 

16 

11 

69% 

1990-1993 

14 

11 

78% 

1994-2000 

40 

30 

75% 

2001-2002 

11 

6 

55% 

Total 

164 

123 

73% 

1981-1982 

14 

7 

50% 

1983-1986 

45 

31 

69% 

2003-2009 

31 

24 

77% 

Total 

90 

62 

69% 

Grand  Total 

254 

185 

73% 

D.  Sources  of  Cost  Growth  Estimates 

Table  B-2  reports  the  sources  of  the  APUC  and  PAUC  estimates  used  in  this  paper. 
Nearly  half  of  the  total  was  taken  from  an  MDAP  cost  growth  database  developed  and 
maintained  by  the  Office  of  Program  Analysis  and  Evaluation  (PA&E)  Resource 
Analysis  deputate.  The  PA&E  cost  growth  database  is  documented  in  a  briefing  by  John 
McCrillis  given  at  the  2003  Annual  DoD  Cost  Analysis  Symposium.3  The  briefing  is 
included  on  the  CD  provided  on  the  inside  back  cover  of  this  paper. 


Table  B-2.  Sources  of  the  APUC  and  PAUC  Growth  Estimates  Used  in  Different  Periods 


Period  (FY) 

PA&E 

P-2722 

RAND 

In  -House 

Total 

1964-1969 

0 

16 

0 

0 

16 

1970-1979 

36 

8 

2 

0 

46 

1980-1989 

45 

0 

4 

1 

51 

1990-1999 

7 

0 

0 

32 

36 

2000-2009 

0 

0 

0 

30 

25 

2010 

0 

0 

0 

3 

3 

Total 

88 

24 

6 

66 

184 

3 

John  McCrillis,  “Cost  Growth  of  Major  Defense  Programs,”  Briefing  (presented  at  the  Department  of 
Defense  Cost  Analysis  Symposium,  Williamsburg,  VA,  January  30,  2003). 
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APUC  and  PAUC  growth  estimates  for  an  additional  24  MDAPs  were  taken  from 
IDA  Paper  P-2722.4  The  provided  CD  includes  the  main  volume  of  P-2722,  as  well  as  an 
Excel  workbook  with  the  data.  The  next  section  of  this  appendix  describes  how  the 
P-5 126  cost  growth  estimates  were  made. 

Communication  from  the  RAND  Corporation  provided  updates  to  the  FY  2015 
SARs  of  estimates  for  six  MDAPs  published  in  a  1996  study5  of  APUC  and  PAUC 
growth  estimates  normalized  to  the  MS  B  baseline. 

Finally,  APUC  and  PAUC  growth  estimates  for  66  MDAPs  were  made  in-house  as 
part  of  the  work  on  this  project.  Fifty-eight  of  the  MDAPs  in  the  PA&E  cost  growth 
database  were  still  ongoing  at  the  time  of  the  final  PA&E  update  (that  is,  when  the 
December  2004  SARs  were  filed).  These  were  updated  with  data  from  the  December 
2015  SARs.  In  addition,  APUC  and  PAUC  growth  estimates  for  MDAPs  that  passed  MS 
B  during  FY  2008-FY  2010  were  made,  again  using  the  December  2015  SARs. 

The  PA&E  estimates  were  constructed  through  a  detailed  examination  of  the  SAR 
variances.  The  IDA  P-2722,  IDA  P-5126,  and  RAND  estimates  were  made  with  data  at  a 
much  more  aggregated  level.  The  methods  used  were  essentially  the  same,  but  it  is 
reasonable  to  assume  that  they  differ  in  detailed  ways  not  captured  by  the  general 
characterization  each  offers  of  the  method  used.  P-2722  did  not  in  all  cases  follow  the 
business  rules  used  in  P-5 126  and  this  paper. 

These  four  sources  use  the  same  definitions  of  the  relevant  cost  terms  and  are  based 
on  SAR  data.  Each  also,  in  most  instances,  measures  cost  growth  from  the  MS  B  baseline 
when  it  is  available  and  reports  quantity  normalized  unit  cost  growth.  Thus,  a  PAUC 
estimate  from,  for  example,  P-2722  means  the  same  thing  as  an  APUC  estimate  from  the 
other  three  sources. 

There  were  several  MDAPs  from  the  1960s  and  1970s  for  which  we  had  two  APUC 
and  PAUC  growth  estimates.  The  decisions  on  which  of  the  alternative  estimates  to  use 
was  entirely  rules-based.  The  PA&E  database  did  not  provide  estimates  for  MDAPs  that 
entered  EMD  during  FY  1965-FY  1969.  The  unit  cost  growth  estimates  used  for 
FY  1965-FY  1969  are  from  P-2722.  In  addition  to  the  SAR  data,  P-2722’s  estimates  in 
many  cases  reflected  other  sources  of  information,  including  material  provided  by  the 
program  office  and  contractors.  For  FY  1970  and  beyond,  we  used  the  PA&E  estimate  in 


4  Karen  Tyson  et  al.,  “The  Effects  of  Management  Initiatives  on  the  Costs  and  Schedules  of  Defense 
Acquisition  Programs,  Vol.  I:  Main  Report,”  IDA  Paper  P-2722  (Alexandria,  VA:  Institute  for  Defense 
Analyses,  1992). 

5  Jeanne  M.  Jarvaise,  Jeffrey  A.  Drezner,  and  Dan  Norton,  “The  Defense  System  Cost  Performance 
Database:  Cost  Growth  Using  Selected  Acquisition  Reports,”  RAND  Report  MR-625-OSD  (Santa 
Monica,  CA:  The  RAND  Corporation,  1996). 
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all  cases  in  which  the  last  SAR  for  the  program  had  been  filed  by  the  time  of  the  final 
update  of  the  PA&E  database  (which  used  the  December  2004  SARs).  In  a  few  cases, 
P-2722  had  a  cost  growth  estimate  for  a  program  not  included  in  the  PA&E  database.  In 
these  instances,  we  used  the  estimate  from  P-2722  if  the  program  was  reported  complete 
in  the  most  recent  SARs  used  in  making  the  estimate;  otherwise,  we  used  the  RAND 
estimate  if  available. 

E.  Computation  of  the  P-5126  (Main  Database)  V  5.3  Estimates 

This  section  briefly  describes  how  the  66  in-house  estimates  were  made.  The 
relevant  data  and  computations  are  in  the  “Data  and  Computations  for  In-house  Cost 
Growth  Estimates. xlsx”  file  on  the  CD. 

1.  RDT&E 

The  SARs  report  fully  configured  units  acquired  with  RDT&E  funds  and  those 
acquired  with  procurement  funds.  Only  the  former  are  used  in  computing  quantity- 
adjusted  RDT&E  cost  growth.  Our  procedure  was  simply  to  compute  the  ratio  of  the 
Current  Estimate  (CE)  of  RDT&E  cost  and  the  baseline  RDT&E  cost  (both  in  program 
base  year  dollars)  and  scale  that  by  the  ratio  of  baseline  quantity  to  CE  quantity.  Suppose, 
for  example,  that  the  number  of  fully  configured  units  purchased  with  RDT&E  funds  has 
increased  from  four  to  five  and  that  CE  RDT&E  cost  is  50  percent  larger  than  the 
baseline  cost.  Our  computation  of  unit  RDT&E  cost  growth  is  then  (4/5)  x  1.5  -  1,  or  20 
percent. 

2.  APUC 

The  method  used  to  nonnalize  APUC  for  quantity  change  depended,  first,  on  the 
extent  to  which  quantity  changed  between  MS  B  and  the  final  SAR  and,  second,  on 
whether  a  useable  estimate  of  the  slope  of  the  learning  curve  was  available. 

a.  No  Quantity  Change  (NQC) 

The  SAR  CE  quantity  was  within  ±1  percent  of  the  MS  B  quantity  for  12  of  the 
MDAPs  for  which  estimates  were  required.  No  quantity  normalization  is  needed  for  these 
programs;  their  APUC  growth  is  computed  by  dividing  the  CE  APUC  in  the  final  SAR 
(or  the  December  2012  SAR  for  an  ongoing  program)  by  the  MS  B  APUC  and 
subtracting  1 .  The  APUC  growth  for  SBIRS-High  also  falls  under  this  heading.  The  total 
number  of  SBIRS-High  satellites  to  be  acquired  decreased  from  five  (at  MS  B)  to  four 
(the  December  2012  SAR).  The  decrease,  however,  was  in  a  satellite  purchased  with 
RDT&E  funds,  and  we  did  not  put  these  on  a  learning  curve.  There  was  no  change  in  the 
number  of  SBIRS-High  satellites  purchased  with  procurement  funds.  Finally,  although 
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the  PAC-3  quantity  change  fell  outside  the  ±1  percent  boundary,  data  limitations  made  it 
necessary  to  compute  the  PAC-3  APUC  growth  as  the  ratio  of  the  CE  and  MS  II  APUCs. 

b.  Defense  Acquisition  Management  Information  Retrieval  System  (DAMIRS) 
Learning  Curve  (DLC) 

The  DoD  contractor  staff  for  DAMIRS  provided  us  with  their  estimates  of  learning 
curve  parameters  that  we  were  able  to  use  to  compute  APUC  growth  for  13  MDAPs  that 
passed  MS  B  during  FY  1989-FY  2001.  We  refer  to  these  as  the  DAMIRS  Learning 
Curve  (DLC)  APUC  growth  estimates.  For  each  of  these,  we  took  the  CE  APUC  growth 
in  program  base  year  dollars  from  the  last  SAR  for  the  program  or  the  December  2015 
SAR  (for  ongoing  programs).  The  task  was  to  normalize  this  APUC  estimate  to  the  MS  B 
quantity,  which  was  done  as  follows: 

•  We  used  the  learning  curve  to  compute  the  recurring  flyaway  cost  at  the  MS  B 
baseline  quantity. 

•  The  CE  estimates  of  RDT&E  and  non-recurring  flyaway  cost  were  taken  from 
the  final  SAR  for  the  program  or  from  the  December  2012  SAR  (for  ongoing 
programs). 

•  Support  costs  paid  for  with  procurement  dollars  are,  for  many  programs, 
primarily  initial  spares  and  support  equipment,  although  other  items  may  also 
fall  into  this  category.  Initial  spares  and  support  equipment  normally  scale  with 
the  number  of  units  of  the  system  purchased.  For  that  reason,  we  used  the  CE 
support  cost  reported  in  the  last  or  most  recent  SAR  scaled  to  the  MS  B  baseline 
quantity. 

c.  Calibrated  Learning  Curve  (CLC) 

Twenty-nine  MDAPs  did  not  have  a  PA&E  estimate  or  estimated  learning  curve 
parameters,  and  their  CE  quantity  was  significantly  different  from  the  MS  B  quantity. 
The  approach  we  used  in  those  cases  rested  on  a  cost  progress  curve  of  the  conventional 
fonn: 

C  =  TQP.  (B-l) 

In  this  expression,  C  is  recurring  flyaway  cost,  T  is  first  unit  cost,  Q  is  cumulative 
production,  and  P  is  the  cost  progress  parameter.  We  solved  this  and  used  the  CE  for 
recurring  flyaway  to  get: 

f  =  CQ~P.  (B-2) 
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This  will  be  referred  to  as  the  calibrated  learning  curve  (CLC)  method.  A  value  of  P  = 
0.94  was  used  for  each  of  the  programs.  From  this  point,  the  computations  were  the  same 
as  those  for  MDAPs  for  which  DAMIRS  staff  provided  the  learning  curve  parameters. 


3.  PAUC 

Quantity  PAUC  is  simply  the  sum  of  quantity  nonnalized  RDT&E  and 
procurement,  divided  by  the  baseline  quantity,  less  1 .  The  baseline  quantity  includes  both 
units  bought  with  RDT&E  funds  and  those  bought  with  procurement  funds. 

4.  Summary 

Table  B-3  provides  an  overview  of  the  number  of  estimates  in  P-5126  made  with 
each  of  the  methods. 


Table  B-3.  Sources  of  the  Quantity  Normalized  Unit  Cost  Growth  Estimates  Used  in 

Different  Periods 


Period  (FY) 

NQC 

DLC 

CLC 

Total 

1989-2001 

5 

13 

21 

39 

2002-2010 

7 

0 

20 

27 

Total 

12 

13 

41 

66 

F.  Comparison  of  the  PA&E  and  CLC/DLC  PAUC  Growth  Estimates 

P-5126  compared  the  PA&E  estimates  for  23  MDAPs  with  estimates  made  using 
the  CLC  and  DLC  methods.6  That  material  is  repeated  here  without  substantial  changes. 

The  obvious  approach  is  to  compare  the  PA&E  PAUC  growth  for  systems  that  have 
been  completed  with  PAUC  growth  for  those  same  systems  computed  using  the  DLC  and 
CLC  methods.  Unfortunately,  there  are  no  MDAPs  that  have  been  completed  and  for 
which  we  have  both  a  PA&E  PAUC  growth  estimate  and  the  data  needed  to  compute  a 
DLC  or  CLC  estimate.  The  best  we  can  do  is  to  examine  the  23  MDAPs  that  passed  MS 
II/B  during  FY  1989-FY  2001  and  for  which  we  have  a  PA&E  PAUC  growth  estimate,  a 
DLC  estimate,  and  a  CLC  estimate. 

The  PA&E  estimates  were  most  recently  updated  with  the  2004  SARs.  The  DLC 
and  CLC  estimates,  in  contrast,  incorporated  more  recent  data — either  the  final  SAR  for 
the  program  or,  for  ongoing  programs,  the  December  2012  SAR.  Consequently,  in  most 
cases  we  would  expect  the  DLC  and  CLC  PAUC  growth  estimates  to  be  larger  than  the 


6  McNicol  and  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process,”  A-7,  A-9. 
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corresponding  PA&E  estimate.  That  is  the  test:  A  method  fails  if  it  yields  estimates  that 
are  “too  often”  and  by  “too  much”  less  than  the  PA&E  estimates.  Clearly,  this  is  a  weak 
test. 

The  relevant  estimates  are  presented  in  Table  B-4.  The  comparison  of  the  PA&E 
estimates  and  CLC  estimates  is  on  the  left,  and  the  comparison  of  the  PA&E  and  DLC 
estimates  is  on  the  right.  The  CLC  estimates  are  larger  than  the  PA&E  estimates  for  17  of 
the  23  MDAPs — in  most  cases,  considerably  larger.  They  are  smaller  in  six  cases  (shaded 
rows).  In  all  but  one  of  these  cases  (Joint  Direct  Attack  Munition,  or  JDAM)  the 
differences  are  absolutely  or  relatively  small.  The  average  of  CLC  PAUC  growth 
estimates  is  77  percent,  in  comparison  to  an  average  of  60  percent  for  the  PA&E 
estimates.  The  DLC  estimates  exhibit  the  same  pattern.  The  average  of  the  DLC 
estimates  is  73  percent,  and  four  of  them  (shaded  rows)  are  less  than  the  PA&E  estimate 
for  the  program,  three  by  a  substantial  amount. 
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Table  B-4.  Comparison  of  PA&E,  CLC,  and  DLC  PAUC  Growth  Estimates  for  23  MDAPs 


Program 

PA&E 

CLC 

Program 

PA&E 

DLC 

Longbow  Apache 

78% 

117% 

Longbow  Apache 

78% 

133% 

F-22 

41% 

71% 

F-22 

41% 

55% 

F/A-18E/F 

6% 

12% 

F/A-18E/F 

6% 

9% 

Bradley  Upgrade 

39% 

54% 

Bradley  Upgrade 

39% 

86% 

MIDS 

30% 

72% 

MIDS 

30% 

68% 

CEC 

48% 

62% 

CEC 

48% 

62% 

H-1  Upgrades 

124% 

192% 

H-1  Upgrades 

124% 

197% 

LPD  17 

43% 

71% 

LPD  17 

43% 

72% 

CH-47F 

147% 

173% 

CH-47F 

147% 

156% 

GMLRS/GMLRS  AW 

125% 

249% 

GMLRS/GMLRS  AW 

125% 

243% 

MH-60S 

62% 

69% 

MH-60S 

62% 

70% 

Tactical  Tomahawk 

24% 

28% 

Tactical  Tomahawk 

24% 

27% 

GBS 

10% 

31% 

GBS 

10% 

33% 

Stryker 

21% 

25% 

Stryker 

21% 

22% 

UH-60M  Black  Hawk 

49% 

62% 

UH-60M  Black  Hawk 

49% 

61% 

WGS 

28% 

55% 

WGS 

28% 

42% 

C-130J 

70% 

84% 

C-130J 

70% 

70% 

JPATS 

43% 

40% 

JPATS 

43% 

44% 

SSN  774 

35% 

33% 

SSN  774 

35% 

37% 

JDAM 

18% 

-10% 

JDAM 

18% 

-13% 

Javelin 

229% 

197% 

Javelin 

229% 

134% 

MH-60R 

95% 

74% 

MH-60R 

95% 

80% 

NAS 

25% 

21% 

NAS 

25% 

1% 

Average 

60% 

77% 

60% 

73% 

Note:  The  PA&E  estimates  were  updated  only  through  the  2004  SARs.  The  CLC  and  DLC  estimates 
incorporate  information  from  the  last  SARforthe  program  or  the  December  2012  SAR  (for  ongoing 
programs). 


B-ll 


Illustrations 


Figures 

Figure  1.  Cumulative  Distribution  of  Time  between  MS  B  and  Cancellation . 4 

Figure  2.  Histogram  of  the  Percentage  of  the  MS  B  Baseline  Quantity  Acquired  by  162 
Completed  MDAPs  that  Passed  MS  B  FY  1965-FY  2009 . 5 

Figure  3.  Number  of  MDAPs  Cancelled  Each  Year,  FY  1965-FY  2016 . 13 

Figure  4.  Number  of  MDAPs  with  a  Completion  Ratio  of  Less  than  75  Percent,  FY 

1965-FY  2016  for  Completed  Programs . 15 

Figure  5.  Scatter  of  Completion  Ratios  and  PAUC  Growth  for  95  MDAPs . 19 

Figure  B-l.  Histogram  of  Quantity  Normalized  PAUC  Growth  from  the  MS  B  Baseline 
for  Completed  MDAPs . B-2 

Tables 

Table  1.  Cancellations,  Total  Programs,  and  Cancellation  Ratios  by  Military  Department 
and  Joint  Programs . 2 

Table  2.  Number  and  Proportion  of  Cancellations  Initiated  by  Different  Levels  of 

Government . 3 

Table  3.  Cohort  Size,  Number  of  Cancellations,  and  Estimated  Probability  of  the  Number 
of  Cancellations  by  Acquisition  Configuration  for  Completed  MDAPs . 7 

Table  4.  Cohort  Size,  Number  of  Programs  with  a  Completion  Ratio  of  Less  than  75 
Percent,  and  Estimated  Probability  of  the  Completion  Ratio  for  Completed 
MDAPs . 8 

Table  5.  PAUC  Growth  for  Completed  Programs  in  Bust  and  Boom  Funding  Climates 
for  a  Given  Acquisition  Policy  and  Process  Configuration . 10 

Table  6.  Cancellations  in  Bust  and  Boom  Funding  Climates  for  Completed  Programs 

Given  Acquisition  Policy  and  Process  Configuration . 10 

Table  7.  Average  Completion  Ratios  in  Bust  and  Boom  Funding  Climates  for  Completed 
Programs  Given  Acquisition  Policy  and  Process  Configuration . 1 1 

Table  8.  Distribution  of  PAUC  Growth  for  Completed  and  Selected  Cancelled 

MDAPs . 17 

Table  9.  Distribution  of  PAUC  Growth  for  Different  Completion  Rates . 19 

Table  B-l.  MDAPs  in  the  Database  Not  Cancelled,  with  an  APUC  and  a  PAUC 

Estimate,  by  Bust/Boom  Time  Periods . B-5 

C-l 


Table  B-2.  Sources  of  the  APUC  and  PAUC  Growth  Estimates  Used  in  Different 

Periods . B-5 

Table  B-3.  Sources  of  the  Quantity  Nonnalized  Unit  Cost  Growth  Estimates  Used  in 

Different  Periods . B-9 

Table  B-4.  Comparison  of  PA&E,  CLC,  and  DLC  PAUC  Growth  Estimates  for  23 

MDAPs . B-ll 


C-2 


References 


Adams,  Gordon,  Paul  Murphy,  and  William  Grey  Rosenau.  Controlling  Weapons  Costs: 
Can  Pentagon  Reforms  Work?  New  York:  Council  on  Economic  Priorities,  1983. 

Augustine,  Nonnan  R.  Augustine ’s  Laws.  Revised  and  expanded  edition.  New  York: 
Viking,  1986. 

Borklund,  C.  W.  The  Department  of  Defense.  New  York:  Frederick  A.  Praeger,  1969. 

Department  of  Defense  Directive  (DoDD)  3200.9,  Initiation  of  Engineering  and 
Operational  Systems  Development,  February  26,  1964. 

DoDD  3200.9,  Initiation  of  Engineering  and  Operational  Systems  Development,  July  1, 
1965  (revised). 

DoDD  5000.1,  The  Defense  Acquisition  System,  May  12,  2003. 

Department  of  Defense  Instruction  (DoDI)  5000.2,  Operation  of  the  Defense  Acquisition 
System,  May  12,  2003. 

Fox,  J.  Ronald.  Defense  Acquisition  Reform,  1969  to  2009:  An  Elusive  Goal. 
Washington,  DC:  U.S.  Army  Center  of  Military  History,  2011. 

Glennan,  Thomas  K.,  Jr.  “Policies  for  Military  Research  and  Development.”  RAND 
Paper  P-3253.  Santa  Monica  CA:  The  RAND  Corporation,  1966. 
https://www.rand.org/content/dam/rand/pubs/papers/2009/P3253.pdf. 

Jarvaise,  Jeanne  M.,  Jeffrey  A.  Drezner,  and  Dan  Norton.  “The  Defense  System  Cost 
Performance  Database:  Cost  Growth  Using  Selected  Acquisition  Reports.”  RAND 
Report  MR-625-OSD.  Santa  Monica,  CA:  The  RAND  Corporation,  1996. 

Jones,  Frank  L.  A  “Hollow  Army  ”  Reappraised:  President  Carter,  Defense  Budgets,  and 
the  Politics  of  Military  Readiness.  Carlisle,  PA:  Strategic  Studies  Institute,  U.S. 
Army  War  College,  2012. 

Kim,  Yool,  Elliot  Axelband,  Abby  Doll,  Mel  Eisman,  Myron  Hura,  Edward  G.  Keating, 
Martin  C.  Libicki,  Bradley  Martin,  Michael  E.  McMahon,  Jerry  M.  Sollinger,  Erin 
York,  Mark  V.  Arena,  Irv  Blickstein,  and  William  Shelton.  “Acquisition  of  Space 
Systems:  Past  Problems  and  Future  Challenges.”  RAND  MG-1 171/7-OSD.  Santa 
Monica  CA:  The  RAND  Corporation,  2015. 

Krepinevich,  Andrew.  “What  Would  Jimmy  Carter  Do?”  http://www.defenseone.com 
/ideas/20 14/03/What  Would  Jimmy  Carter-do-Defense-One.htm. 

McCrillis,  John.  “Cost  Growth  of  Major  Defense  Programs.”  Briefing.  Presented  at 

Department  of  Defense  Cost  Analysis  Symposium,  Williamsburg,  VA,  January  30, 
2003. 


D-l 


McNicol,  David  L.  Cost  Growth  in  Major  Weapon  Procurement  Programs .  2nd  ed. 
Alexandria,  VA:  Institute  for  Defense  Analyses,  2004. 

McNicol,  David  L.,  Sarah  K.  Bums,  and  Linda  Wu.  “Evidence  on  the  Effect  of  DoD 
Acquisition  Policy  and  Process  and  Funding  Climate  on  Cancellation  of  Major 
Defense  Acquisition  Programs.”  IDA  Paper  P-5218.  Alexandria,  VA:  Institute  for 
Defense  Analyses,  May  2015. 

McNicol,  David  L.,  David  M.  Tate,  Sarah  K.  Bums,  and  Linda  Wu.  “Further  Evidence  on 
the  Effect  of  Acquisition  Policy  on  Cost  Growth  of  Major  Defense  Acquisition 
Programs.”  IDA  Paper  P-5330  (Revised).  Alexandria,  VA:  Institute  for  Defense 
Analyses,  August  2016. 

McNicol,  David  L.,  and  Linda  Wu.  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy 
and  Process  on  Cost  Growth  of  Major  Defense  Acquisition  Programs.”  IDA  Paper 
P-5126.  Alexandria,  VA:  Institute  for  Defense  Analyses,  September  2014. 

Murdock,  Clark  A.  Defense  Policy  Formation.  Albany,  NY:  State  University  of  New 
York  Press,  1974. 

O’Neil,  William  D.,  and  Gene  H.  Porter.  “What  to  Buy?  The  Role  of  Director  of  Defense 
Research  and  Engineering  (DDR&E) — Lessons  from  the  1970s.”  IDA  Paper 
P-4675.  Alexandria,  VA:  Institute  for  Defense  Analyses,  January  2011. 

Porter,  Gene,  Brian  Gladstone  C.  Vance  Gordon  Nicholas  Karvonides  R.  Royce  Kneece, 
Jr.  Jay  Mandelbaum,  and  William  D.  O’Neil.  “The  Major  Causes  of  Cost  Growth  in 
Defense  Acquisition:  Volume  I-Executive  Summary.”  IDA  Paper  P-4513. 
Alexandria,  VA:  Institute  for  Defense  Analyses,  December  2009. 

Selected  Acquisition  Report:  Global  Broadcast  System.  Defense  Acquisition  Management 
Infonnation  Retrieval  (DAMIR).  December  2003. 

Srull,  Donald,  ed.  The  Cost  Analysis  Improvement  Group:  A  History.  McLean,  VA: 
Logistics  Management  Institute  (now  LMI),  1998. 

Tyson,  Karen  W.,  Neang  I.  Om,  D.  Calvin  Gogerty,  and  J.  Richard  Nelson.  “The  Effects 
of  Management  Initiatives  on  the  Costs  and  Schedules  of  Defense  Acquisition 
Programs,  Vol.  I:  Main  Report.”  IDA  Paper  P-2722.  Alexandria,  VA:  Institute  for 
Defense  Analyses,  1992. 

United  States  Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense 

(Comptroller).  National  Defense  Budget  Estimates  for  FY  2014.  Washington,  DC: 
Department  of  Defense,  May  2013. 

Whittle,  James  R.  The  Dream  Machine.  New  York:  Simon  and  Schuster,  2010. 

Younossi,  Obaid,  Mark  A.  Lorell,  Kevin  Brancato,  Cynthia  R.  Cook,  Mel  Eisman, 
Bernard  Fox,  John  C.  Graser,  Yool  Kim,  Robert  S.  Leonard,  Shari  Lawrence 
Pfleeger,  and  Jerry  M.  Sollinger.  “Improving  the  Cost  Estimation  of  Space  Systems: 
Past  Lessons  and  Future  Recommendations.”  MG-690-AF.  Santa  Monica,  CA:  The 
RAND  Corporation,  2008. 


D-2 


Abbreviations 


ACAT 

Acquisition  Category 

A-D 

Anderson-Darling  Test 

AHIP 

Anny  Helicopter  Improvement  Program 

ANOVA 

Analysis  of  Variance 

APUC 

Average  Procurement  Unit  Cost 

AR 

Acquisition  Reform 

ATM 

Asynchronous  Transfer  Mode 

BA 

Budget  Authority 

CAIG 

Cost  Analysis  Improvement  Group 

CAPE 

Cost  Assessment  and  Program  Evaluation 

CD 

Compact  Disc 

CE 

Current  Estimate 

CLC 

Calibrated  Learning  Curve 

CPIF 

Cost  Plus  Incentive  Fee 

CY 

Calendar  Year 

DAB 

Defense  Acquisition  Board 

DAMIR 

Defense  Acquisition  Management  Infonnation  Retrieval 

DCP 

Development  Concept  Paper/Decision  Coordinating  Paper 

DDR&E 

Director,  Defense  Research  and  Engineering 

DLC 

DAMIRS  Learning  Curve 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

DSARC 

Defense  Systems  Acquisition  Review  Council 

EMD 

Engineering  and  Manufacturing  Development 

FET 

Fisher’s  Exact  Test 

FFP 

Finn  Fixed  Price 

FPIF 

Fixed  Price  Incentive  Fee 

FY 

Fiscal  Year 

GBS 

Global  Broadcast  System 

GRH 

Gramm-Rudman-Hollings 

E-l 


IDA 

IFF 

IP 

JDAM 

K-S 

MAIS 

MDA 

MDAP 

MILPERS 

MS 

M-WU 

NQC 

O&M 

OASD(SA) 

OLS 

OSA 

OSD 

PA&E 

PAUC 

P-C 

PPBS 

RDT&E 

SAR 

SBIRS-High 

TPP 

US 

USD(AT&L) 

VCJCS 

WSARA 


Institute  for  Defense  Analyses 

Iraqi  Freedom  Funds 

Internet  Protocol 

Joint  Direct  Attack  Munition 

Kolmogorov-Smimov  Test 

Major  Automated  Infonnation  System 

Milestone  Decision  Authority 

Major  Defense  Acquisition  Program 

Military  Personnel 

Milestone 

Mann-Whitney  U  Test 
No  Quantity  Change 
Operations  and  Maintenance 

Office  of  the  Assistant  Secretary  of  Defense  for  Systems  Analysis 

Ordinary  Least  Squares 

Office  of  Systems  Analysis 

Office  of  the  Secretary  of  Defense 

Program  Analysis  and  Evaluation 

Program  Acquisition  Unit  Cost 

Post-Carlucci 

Planning,  Programming,  and  Budgeting  System 
Research,  Development,  Test  and  Evaluation 
Selected  Acquisition  Report 
Space-Based  Infrared  Satellite-High 
Total  Package  Procurement 
United  States 

Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 

Vice  Chairman,  Joint  Chiefs  of  Staff 

Weapon  Systems  Acquisition  Reform  Act  of  2009 


E-2 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of 
information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188), 
1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  V A  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any 
penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE 

xx-03-2017  Final 


4.  TITLE  AND  SUBTITLE 

Influences  on  the  Timing  and  Frequency  of  Cancellations  and  Truncations  of 
Major  Defense  Acquisition  Programs 


3.  DATES  COVERED  ( From  -  To) 
Jul  2016  -  Feb  2017 


5a.  CONTRACT  NUMBER 

HQ0034- 1 4-D-000 1 
5b.  GRANT  NUMBER 


6.  AUTHOR(S) 

McNicol,  David,  L. 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


AY-7-4153 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Institute  for  Defense  Analyses 
4850  Mark  Center  Drive 
Alexandria,  VA  22311-1882 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Deputy  Director  for  Root  Cause  Analyses  (Acting) 

Office  of  the  Assistant  Secretary  of  Defense  for  Acquisition 
Room  3C889A,  The  Pentagon 
3620  Defense  Pentagon 
Washington,  DC  20301 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

IDA  Paper  P-8280 
Log  #:  H  16-001313  (Main  Paper) 
Log  #:  H  17-000067  (Data  CD) 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

OSD-PARCA 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


13.  SUPPLEMENTARY  NOTES 

CD-ROM  included  on  the  inside  back  cover. 


14.  ABSTRACT 

A  previous  paper  in  this  series  examined  whether  changes  in  acquisition  policy  and  funding  climate  had  statistically  discernible 
associations  with  cancellations  of  Major  Defense  Acquisition  Programs  (MDAPs).  It  did  not  provide  any  statistical  analysis  of  truncations 
— that  is,  MDAPs  that  bought  fewer  units  of  a  system  than  was  intended  at  Milestone  (MS)  B.  This  paper  extends  the  earlier  one;  it 
provides  parallel  analyses  of  both  cancellations  and  truncations,  has  more  observations  (especially  on  truncations),  and  provides  an 
improved  analysis  of  cancellations.  The  paper  finds  that  cancellations  of  MDAPs  are  concentrated  in  the  two  periods  during  which  DoD 
procurement  funding  was  declining,  often  sharply.  With  one  possible  exception,  cancellations  and  truncations  are  (1)  not  associated  with 
the  acquisition  policy  and  process  configuration  in  place  when  they  passed  MS  B,  given  funding  climate;  and  (2)  not  consistently 
associated  with  the  funding  climate  prevailing  at  MS  B.  given  the  acquisition  policy  and  process  configuration.  The  paper  also  finds  that, 
while  cancelled  programs  on  average  had  higher  cost  growth  than  programs  not  cancelled,  most  programs  with  unit  cost  growth  of  more 
than  50  percent  were  not  cancelled. 


15.  SUBJECT  TERMS 

Acquisition  reform;  acquisition  policy;  acquisition  process;  cancellation;  truncation;  completion  ratio;  termination;  cost  growth; 
APUC  growth;  PAUC  growth;  funding  climate 


16.  SECURITY  CLASSIFICATION  OF: 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

Unclassified 

Unclassified 

Unclassified 

17.  LIMITATION  OF 
ABSTRACT 

Same  as  Report 


18.  NUMBER  19a.  NAME  OF  RESPONSIBLE  PERSON 

0F  Cadman,  David 

PAGES  - ' - 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

60  (703)  697-3733 


Standard  Form  298  (Rev.  8/98) 
Prescribed  by  ANSI  Std.  Z39.18 


